Image processing device, image processing method, imaging device, and recording medium for exclusively performing adjustment processing or viewpoint change processing

ABSTRACT

An image processing device includes at least one processor that acquires a plurality of viewpoint images, and selectively applies a plurality of different image processings to image data based on the plurality of viewpoint images. The plurality of different image processings includes an adjustment process of adjusting a perceived resolution of an image, the adjustment process including a shift synthesis process of relatively shifting the plurality of viewpoint images to synthesize the plurality of viewpoint images that are relatively shifted, and a viewpoint change process of changing a viewpoint by changing a weighting coefficient when the plurality of viewpoint images are synthesized. In addition, the at least one processor sets the image processing by the image processing unit, exclusively setting one of the plurality of different image processings and a parameter of the image processing set by the setting unit to be applied to the image data.

This application claims the benefit of Japanese Patent Application No.2016-174951, filed Sep. 7, 2016, and Japanese Patent Application No.2015-232925, filed Nov. 30, 2015, which are hereby incorporated byreference wherein in their entireties.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to image processing technology using aviewpoint image.

Description of the Related Art

Image processing using a plurality of parallax images (viewpoint images)corresponding to a plurality of viewpoints including the same objectimage obtained by photographing is proposed. In Japanese PatentLaid-Open No. 2012-186790, technology for generating a captured image ofany (virtual) viewpoint according to adjustment of a synthesis ratio ofcaptured images at multiple viewpoints from the captured images isdisclosed. In Japanese Patent Laid-Open No. 2014-228586, technology forgenerating an image refocused on a virtual image plane by relativelyshifting and synthesizing a plurality of parallax images acquired froman imaging element in which a plurality of photoelectric conversionunits are allocated to one microlens is disclosed.

Both Japanese Patent Laid-Open No. 2012-186970 and Japanese PatentLaid-Open No. 2014-228586 only disclose, however, an image processingdevice in which one type of image processing is enabled using aplurality of viewpoint images.

SUMMARY OF THE INVENTION

The present invention provides an image processing device including aconfiguration in which a plurality of types of image processing are ableto be applied using a plurality of viewpoint images.

In one aspect of the present invention, an image processing deviceincludes an acquisition unit configured to acquire a plurality ofviewpoint images, an image processing unit configured to perform imageprocessing on image data based on the plurality of viewpoint images, anda setting unit configured to set the image processing of the imageprocessing unit, wherein the image processing unit is able toselectively apply a plurality of different image processings to theimage data, and wherein the setting unit is able to set whether to applyeach of the plurality of image processings to the image data and aparameter of image processing to be applied to the image data.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a basic configuration of an imageprocessing device according to the present invention.

FIG. 2 is a block diagram illustrating a configuration example of animaging device according to an embodiment of the present invention.

FIG. 3 is a schematic diagram of a pixel array of an imaging element inan embodiment of the present invention.

FIGS. 4A and 4B are a schematic plan view and a schematiccross-sectional view, respectively, of the imaging element.

FIGS. 5A and 5B are diagrams illustrating the correspondence between apixel of the imaging element and a pupil division area.

FIGS. 6A and 6B are diagrams illustrating a relationship between a pupildivision, a defocus amount, and an image shift amount.

FIG. 7 is a flowchart of a process of adjusting a perceived resolutionin an embodiment of the present invention.

FIGS. 8A to 8C are diagrams illustrating shading due to pupil shifts.

FIG. 9 is a diagram illustrating an example of a captured image.

FIG. 10 is a diagram illustrating an example of an imaging contrastdistribution of the captured image.

FIG. 11 is a diagram illustrating an example of a first viewpointcontrast distribution of a first viewpoint image.

FIG. 12 is a diagram illustrating an example of a second viewpointcontrast distribution of a second viewpoint image.

FIGS. 13A to 13C are schematic relationship diagrams of a parallaxbetween viewpoint images and perspective conflict.

FIG. 14 is a diagram illustrating contrast difference amountdistributions of a first viewpoint image and a second viewpoint image.

FIG. 15 is a diagram illustrating an example of a contrast distribution.

FIG. 16 is a diagram illustrating examples of image shift amountdistributions of the first viewpoint image and the second viewpointimage.

FIG. 17 is a diagram illustrating an image shift difference amountdistribution from a predetermined shift amount.

FIG. 18 is a diagram illustrating a (crosstalk correction) process ofsharpening a parallax between viewpoint images.

FIG. 19 is a schematic explanatory diagram of refocusing by a shiftsynthesis process.

FIG. 20 is a schematic explanatory diagram of a range in whichrefocusing by the shift synthesis process is possible.

FIGS. 21A and 21B are diagrams illustrating refocused images in theconventional technology and technology of the present embodiment.

FIG. 22 is a diagram illustrating an example of an image in whichforeground blur fogging for a main object occurs.

FIGS. 23A and 23B are explanatory diagrams of an effect of foregroundblur fogging correction.

FIG. 24 is a flowchart of a ghost reduction process in an embodiment ofthe present invention.

FIG. 25 is a flowchart of an operation of an image processing device inan embodiment of the present invention.

DESCRIPTION OF THE EMBODIMENTS

An overview of the invention will be described before description ofeach embodiment of the present invention. FIG. 1 is a conceptualexplanatory diagram illustrating a representative example of a basicconfiguration of an image processing device according to the presentinvention. One or more of the functional blocks shown in FIG. 1 may beimplemented by hardware, such as an application specific integratedcircuit (ASIC) or a programmable logic array (PLA), or may beimplemented by a programmable processor, such as a central processingunit (CPU) or a microprocessing unit (MPU) executing software. One ormore of the functional blocks may also be implemented using acombination of software and hardware. Accordingly, in the followingdescription, even if different functional blocks are described as anoperating subject, the same hardware can be implemented as the subject.

An input unit 101 acquires data of a plurality of viewpoint images. Acontrol unit 104 controls units including the input unit 101. Accordingto an instruction of the control unit 104, a first processing unit 102determines relative coordinates (a shift amount) of each viewpoint imagefor acquisition of distribution information, a smoothing process, andfor a shift synthesis performed by a synthesis unit 105, to be describedbelow, with respect to the acquired plurality of viewpoint images. Asecond processing unit 103 determines a synthesis ratio of the pluralityof viewpoint images in the viewpoint image synthesis process forreproducing images from various viewpoints. Also, the second processingunit 103 detects a ghost amount (a ghost component) to reduce a ghost(unnecessary light) occurring in a synthesized image. The synthesis unit105 performs the smoothing process on the plurality of viewpoint imagesaccording to a setting of image processing to be applied. The synthesisunit 105 further performs the synthesis process on the images on thebasis of the shift amount and the synthesis ratio, subtracts the ghostcomponent detected by the second processing unit 103 from thesynthesized image, and transmits the resulting synthesized image to anoutput unit 106. The output unit 106 processes image data according toan apparatus or a device of an output destination and outputs theprocessed image data. Image processing to be executed by the firstprocessing unit 102, the second processing unit 103, and the synthesisunit 105 will be described in detail using specific examples inembodiments.

First Embodiment

Hereafter, the first embodiment of the present invention will bedescribed in detail. FIG. 2 is a block diagram illustrating aconfiguration example in which the image processing device of thepresent embodiment is applied to the imaging device. In the presentembodiment, an example of a digital camera 100 capable of recordingviewpoint image data will be described.

A photographing lens 230 serves as a plurality of optical membersconstituting an imaging optical system and includes a lens and anaperture 240. An imaging element 110 converts an optical image of anobject formed through the photographing lens 230 into an electricalsignal through photoelectric conversion. An analog-to-digital (A/D)converter 120 converts an analog signal output of the imaging element110 into a digital signal. In the present embodiment, the photographinglens 230, the imaging element 110, and the A/D converter 120 areincluded in the input unit 101 of FIG. 1.

The image processing unit 130 performs predetermined de-mosaicprocessing or color conversion processing, or the like, on data from theA/D converter 120 or data recorded on a random access memory (RAM). Theimage processing unit 130 is included in the first and second processingunits 102 and 103 of FIG. 1 and the synthesis unit 105. That is, in thepresent embodiment, the image processing unit 130 and a CPU 170 performrefocus processing and sharpness/unsharpness control, as will bedescribed below. Further, a process of detecting a ghost on the basis ofa difference among a plurality of viewpoint images to control aninfluence of a ghost appearing in an image and correcting image data toreduce an influence of the ghost is performed. Output data of the A/Dconverter 120 is written in a RAM 190 via the image processing unit 130and a camera signal processing unit 140 or directly via the camerasignal processing unit 140.

The CPU 170 is a central unit which controls the overall system andcorresponds to the control unit 104 of FIG. 1. The CPU 170 performsvarious types of processes that are shown and described herein byreading and executing a program stored in a read only memory (ROM) 180.An operation unit 210 includes an operation member, such as a shutterrelease switch, and outputs an operation instruction signal of a user tothe CPU 170. For example, according to an operation instruction of afirst stage of the shutter release switch, an imaging system controlunit 205 controls the driving of the aperture 240 or the lens. AF(autofocus) processing, AE (automatic exposure) processing, AWB (autowhite balance) processing, EF (flash pre-emission) processing, or thelike starts. According to an operation instruction of a second stage ofthe shutter release switch, the A/D converter 120 converts an analogoutput signal of the imaging element 110 into a digital image signal,and the image processing unit 130 and the camera signal processing unit140 process the digital image signal. The camera signal processing unit140 acquires an output of the image processing unit 130 or image datafrom the RAM 190, and executes a developing process using a calculation.For example, the camera signal processing unit 140 performs de-mosaicprocessing, a defect correction process or a shading correction processspecific to the imaging element 110, a process of correcting a blacklevel, or the like, white balance processing, a gamma correctionprocess, a color conversion process, and a noise reduction process.Also, the camera signal processing unit 140 performs a process ofcompressing image data and the like, and outputs the processed data to amedia interface (I/F) unit 150. The media I/F unit 150 executes arecording process of writing image data in a recording medium 160.

The RAM 190 is a memory that stores data of a still image or a movingimage after photographing, and has a storage capacity sufficient forstoring a predetermined number of still images or moving images of apredetermined time. An image display unit 220 is constituted of a thinfilm transistor (TFT) type liquid crystal display, or the like, andimage display is performed according to display image data written inthe RAM 190. Here, the image display unit 220 sequentially displayscaptured images, thereby implementing a live view function.

FIG. 3 is a schematic diagram illustrating a pixel array of the imagingelement 110 and illustrates a pixel array of a two-dimensionalcomplementary metal-oxide-semiconductor (CMOS) image sensor. A directionperpendicular to the paper surface of FIG. 3 is defined as az-direction, a right/left direction is defined as an x-direction, and anup/down direction is defined as a y-direction. Imaging pixels are shownin a range of four rows and four columns and a sub-pixel array is shownin a range of four rows and eight columns. A pixel group 200 of two rowsand two columns, illustrated in FIG. 3, includes pixels 200R, 200G, and200B. The pixel 200R having spectral sensitivity of red (R) is locatedat an upper-left position in FIG. 3, the pixel 200G having spectralsensitivity of green (G) is located at an upper-right position and alower-left position, and the pixel 200B having spectral sensitivity ofblue (B) is located at a lower-right position. Further, each pixel 200R,200G, and 200B is constituted of a first sub-pixel 201 and a secondsub-pixel 202 arrayed in one row and two columns.

In the imaging element 110, it is possible to acquire a captured imagesignal and a viewpoint image signal by arranging a large number ofimaging pixels of 4 rows and 4 columns (sub-pixels of 4 rows and 8columns) illustrated in FIG. 3 on an imaging plane. A plan view of onepixel 200G in the imaging element 110 when viewed from a light receivingplane side (a positive z-direction) is illustrated in FIG. 4A. In FIG.4A, the direction perpendicular to the paper surface is defined as thez-direction, the left/right direction is defined as the x-direction, theup/down direction is defined as the y-direction. The front side isdefined as the positive z-direction, the right direction is defined asthe positive x-direction, and the up direction is defined as thepositive y-direction. A cross-sectional view of the a-a cross section ofFIG. 4A viewed from the negative y-direction is illustrated in FIG. 4B.In FIG. 4B, the direction perpendicular to the paper surface is they-direction, the left-right direction is the x-direction, and theup/down direction is the z-direction

In the pixel 200G, a microlens 305 for focusing incident light on thelight receiving plane side is formed. The photoelectric conversion unit300 is divided into N_(H) divisions in the x-direction and divided intoN_(V) divisions in the y-direction. Although N_(H)=2, N_(V)=1, and twophotoelectric conversion units 301 and 302 are formed in the example ofFIGS. 4A and 4B, the number of divisions can be arbitrarily set. Thephotoelectric conversion units 301 and 302 correspond to the firstsub-pixel 201 and the second sub-pixel 202, respectively. Thephotoelectric conversion units 301 and 302 have a pin structurephotodiode in which an intrinsic layer is sandwiched between a p-typelayer and an n-type layer, or a pn junction photodiode in which theintrinsic layer is omitted.

In each pixel, a color filter 306 between the microlens 305 and thephotoelectric conversion units 301 and 302 is formed. When necessary,spectral transmittance of the color filter is changed for each sub-pixelor the color filter is omitted. Light incident on the pixel 200G isfocused by the microlens 305 and the photoelectric conversion units 301and 302 receive light after spectral separation in the photoelectricconversion units 301 and 302. In the photoelectric conversion units 301and 302, pairs of electrons and holes are generated according to anamount of received light and are separated by a depletion layer, andnegatively charged electrons are accumulated in an n-type layer (notillustrated). On the other hand, the holes are discharged outside animaging element through a p-type layer connected to a constant voltagesource (not illustrated). The electrons accumulated in the n-type layersof the photoelectric conversion units 301 and 302 are transferred to astatic capacitance unit (FD) via a transfer gate and are converted intoa voltage signal, and the voltage signal is output as a pixel signal.

FIG. 5A is a diagram illustrating a correspondence relationship betweenthe pixel structure and the pupil division illustrated in FIGS. 4A and4B. A cross-sectional view of a cross section of a pixel structure takenalong the line a-a when viewed from the positive y-direction isillustrated in the lower portion of FIG. 5A, and a view of an exit pupilplane of an image forming optical system (see an exit pupil 410) whenviewed from the negative z-direction is illustrated in the upper portionof FIG. 5A. In FIG. 5A, the x-axis and the y-axis obtained by invertingthe state illustrated in FIG. 4B are illustrated in the cross-sectionalview of the pixel structure to take the correspondence with thecoordinate axes of the exit pupil plane.

The first pupil part area 401 corresponding to the first sub-pixel 201is generally set to be in a conjugate relationship by the microlens 305with respect to a light receiving plane of the photoelectric conversionunit 301 having a center of gravity biased in the negative x-direction.That is, the first pupil part area 401 represents a pupil area capableof being received by the first sub-pixel 201, and has a center ofgravity biased in the positive x-direction on an exit pupil plane. Inaddition, the second pupil part area 402 corresponding to the secondsub-pixel 202 is generally set to be in a conjugate relationship by themicrolens 305 with respect to a light receiving plane of thephotoelectric conversion unit 302 having a center of gravity biased inthe positive x-direction. The second pupil part area 402 represents apupil area capable of being received by the second sub-pixel 202, andhas a center of gravity biased in the negative x-direction on an exitpupil plane.

An area 400 illustrated in FIG. 5A is a pupil area in which light can bereceived by the entire pixel 200G when the photoelectric conversion unit301 and the photoelectric conversion unit 302 (the first sub-pixel 201and the second sub-pixel 202) are combined. A correspondencerelationship between the imaging element and the pupil division isillustrated in a schematic diagram of FIG. 6A. Light beams passingthrough the first pupil part area 401 and the second pupil part area 402are incident on pixels of the imaging element at different angles.Incident light on the imaging plane 500 is received by each of thephotoelectric conversion units 301 and 302 of N_(H)(=2)×N_(V) (=1)divisions and each of the photoelectric conversion unit 301 and 302converts light into an electrical signal.

The image processing unit 130 generates a first viewpoint image bycollecting light reception signals of the first sub-pixels 201 of pixelunits, and generates a second viewpoint image by collecting lightreception signals of the second sub-pixels 202 in focus detection. Also,the image processing unit 130 generates an imaging signal ofpredetermined resolution by adding a signal of the first sub-pixel 201to a signal of the second sub-pixel 202 to output captured image data.Relationships between image shift amounts and defocus amounts of thefirst viewpoint image and the second viewpoint image calculated forfocus detection during imaging, a process of adjusting perceivedresolution characteristics in the present embodiment, or the like willbe described with reference to FIG. 6B.

In FIG. 6B, an imaging element (not illustrated) is arranged on theimaging plane 500. The exit pupil 410 of the image forming opticalsystem is divided into two areas of a first pupil part area 401 and asecond pupil part area 402. In the defocus amount d, a magnitude of thedefocus amount d indicates a distance from an image formation positionof an object image to the imaging plane 500. A direction in which anegative sign (d<0) is set in a front focus state, in which the imageformation position of the object image is at the object side rather thanthe imaging plane 500, and a positive sign (d>0) is set in a rear focusstate opposite to the front focus state, is defined. In the focus state,in which the image formation position of the object image is in theimaging plane (the focus position), d=0. An example in which theposition of the object 601 illustrated in FIG. 6B is a positioncorresponding to the focus state (d=0) is shown, and an example in whichthe position of the object 602 is a position corresponding to the frontfocus state (d<0) is shown. Hereinafter, the front focus state (d<0) andthe rear focus state (d>0) are collectively referred to as the defocusstate (|d|>0).

In the front focus state (d<0), a light beam passing through the firstpupil part area 401 (or the second pupil part area 402) among lightbeams from the object 602 extends to a width Γ1 (or Γ2) around acenter-of-gravity position G1 (or G2) of the light beam after beingfocused once. In this case, a blurred image occurs in the imaging plane500. For the blurred image, light is received by the first sub-pixel 201(or the second sub-pixel 202) constituting each pixel arrayed in theimaging element and the first viewpoint image (or the second viewpointimage) is generated. Consequently, the first viewpoint image (or thesecond viewpoint image) is detected as an object image (a blurred image)having the width Γ1 (or Γ2) at the center-of-gravity position G1 (or G2)on the imaging plane 500. The width Γ1 (or Γ2) of the object imagegenerally increases in proportion to an increase of the magnitude |d| ofthe defocus amount d. Likewise, when an image shift amount of the objectimage between the first viewpoint image and the second viewpoint imageis denoted by “p”, a magnitude |p| of the shift image amount p increasesaccording to an increase of the magnitude |d| of the defocus amount d.For example, the image shift amount p is defined as a difference “G1-G2”between center-of-gravity positions of light beams, and the magnitude|p| increases generally in proportion to an increase of |d|. Also,although the image shift direction of the object image between the firstviewpoint image and the second viewpoint image in the rear focus state(d>0) is opposite to that in the front focus state, there is a similartrend. In the present embodiment, when magnitudes of defocus amounts dof the first viewpoint image and the second viewpoint image or animaging signal obtained by adding the first viewpoint image and thesecond viewpoint image increase, the magnitude |p| of the image shiftamount p between the first viewpoint image and the second viewpointimage increases.

Next, various types of image processing distinctive to the presentembodiment to be performed by the image processing unit 130 will bedescribed. In the present embodiment, some or all of a process ofadjusting a perceived resolution, a process of generating an image inwhich a viewpoint is changed, and a process of reducing a ghost(unnecessary light) occurring in an image are performed as a process tobe performed on the image using a plurality of viewpoint images.

For example, the process of adjusting a perceived resolution will bedescribed. Here, the perceived resolution represents the overallimpression of sharpness from the resolution of an object image, an edge,blurriness, etc. in an image. That is, the process of adjusting aperceived resolution in the present embodiment includes a process ofadjusting at least one of parameters of the image. For example, it isconceivable to increase the sharpness by the adjustment of luminance,saturation, and hue of the image, and the process of adjusting theperceived resolution is not limited to the above-described process, andother processes may be included.

Hereafter, a process necessary for each viewpoint image acquired in thepresent invention for adjusting the perceived resolution will bedescribed. First, the properties of a pixel signal obtained by a lightbeam incident on each pixel (the photoelectric conversion element) ofthe imaging element 110 in the present embodiment will be described.

If light is incident on the microlens 305 formed in each pixel 200R,200G, 200B, the incident light is focused at a focus position by themicrolens 305. Because of an influence of diffraction due to the wavenature of light, however, the diameter of a light focus spot cannot beless than a diffraction limit A, and has a finite magnitude. While alight receiving plane size of the photoelectric conversion unit 301, 302is about 1 to 2 μm, a light focus size of the microlens 305 is about 1μm. Thus, the first and second pupil part areas 401 and 402 in aconjugate relationship with the light receiving plane of thephotoelectric conversion unit 301, 302 via the microlens 305 are notclearly divided due to a diffraction blur, and have a light receivingrate distribution (a pupil intensity distribution) depending upon anincident angle of light.

FIG. 5B illustrates an example of the light receiving rate distribution(the pupil intensity distribution) depending upon the incident angle oflight. The horizontal axis represents pupil coordinates and the verticalaxis represents a light receiving rate. A graph line L1, indicated bythe solid line in FIG. 5B, represents a pupil intensity distributionalong the x-axis of the first pupil part area 401 of FIG. 5B. The lightreceiving rate indicated by the graph line L1 rapidly increases from aleft end to reach a peak, and then gradually decreases and reaches aright end after the rate of change decreases. Also, a graph line L2,indicated by the dotted line in FIG. 5B, represents a pupil intensitydistribution along the x-axis of the second pupil part area 402. Incontrast to the graph line L1 (symmetrically with respect to the rightand left sides), the light receiving rate indicated by the graph line L2rapidly increases from a right end to reach a peak, and then graduallydecreases and reaches a left end after the rate of change decreases. Asillustrated, it can be seen that the pupil is gradually divided. Evenwhen a shift synthesis (refocusing) process of performing a relativeshift between viewpoint images and synthesizing shifted images isperformed on a plurality of viewpoint images obtained in a state inwhich the gradual pupil division is performed, the effect is reducedbecause clear pupil division has not been performed initially.Therefore, in the present embodiment, a crosstalk correction (smoothing)process for emphasizing the pupil division (pupil separation) among aplurality of viewpoint images is performed for a process of adjustingthe perceived resolution.

Refocus Process and Sharpness/Unsharpness Control

In the present embodiment, a refocus process of re-correcting a focusposition after imaging for a captured image using relationships betweendefocus amounts and image shift amounts of the first viewpoint image andthe second viewpoint image is performed. In the present embodiment, aprocess in which the following two processes are combined is performedas the refocus process. One process is a refocus process based on ashift synthesis process using the first viewpoint image and the secondviewpoint image. The other process is sharpness/unsharpness control foradaptively controlling an area having a high degree of sharpness and anarea having a high degree of blurriness by sharpening and smoothingaccording to an image shift difference amount distribution. Theembodiment is not limited thereto, however, and only one of theabove-described refocus process and sharpness/unsharpness control may beapplied to the image. If one process is performed, it is only necessaryto omit a step related to only the other process.

FIG. 7 is a flowchart illustrating an overview of a flow of the refocusprocess and the sharpness/unsharpness control. The process of FIG. 7 isexecuted by the CPU 170 and the image processing unit 130 that are imageprocessing means of the present embodiment.

Multi-Viewpoint Image and Capturing Image

In step S1 of FIG. 7, a plurality of viewpoint images are generated fromLF data (an input image) acquired by the imaging element of the presentembodiment for each different pupil part area of the image formingoptical system. For an image displayed for the setting of a parameter ofeach image processing, a process of generating a captured image (asynthesized image) according to a pupil area obtained by synthesizingdifferent pupil part areas of the image forming optical system isperformed. Although the LF data (the input image) acquired by theimaging element in step S1 is input, LF data (the input image) saved ina recording medium through photographing by the imaging element inadvance may be read and used.

Next, in step S1, the first viewpoint image and the second viewpointimage (first to N_(LF) ^(th) viewpoint images) are generated for eachdifferent pupil part area of the image forming optical system. The LFdata (the input image) is denoted by LF. Also, a k^(th) sub-pixel signalis provided by designating a sub-pixel signal that is i_(s) ^(th)(1≤i_(s)≤N_(x)) in the column direction and is j_(s) ^(th)(1≤j_(s)≤N_(y)) in the row direction as k=N_(x)(j_(s)−1)+i_(s)(1≤k≤N_(LF)) within each pixel signal of the LF data. A k^(th) viewpointimage I_(k)(j, i) that is i^(th) in the column direction and is j^(th)in the row direction corresponding to the k^(th) pupil part area of theimage forming optical system is generated by Formula (1).I _(s)(j,i)=I _(N) _(x) _((j) _(s) _(−1)+i) _(s) (j,i)=LF(N _(y)(j−1+j_(s) ,N _(x)(i−1)+i _(s)).  (1)

In the present embodiment, an example of two divisions in the x-axisdirection for N_(x)=2, N_(y)=1, and N_(LF)=2 is shown. A process ofselecting a signal of a specific sub-pixel from the first sub-pixel 201and the second sub-pixel 202 obtained by dividing the pixel into twosub-pixels in the x-direction (first to N_(LF) ^(th) sub-pixels obtainedby dividing the pixel into N_(x)×N_(y) sub-pixels) for each pixel fromthe LF data (the input data) corresponding to the pixel arrayillustrated in FIG. 3 is executed. The first viewpoint image and thesecond viewpoint image (the N_(LF) ^(th) viewpoint image) that are RGBsignals of a Bayer array having a resolution of the number of pixels Ncorresponding to a specific pupil part area of the first pupil part area401 and the second pupil part area 402 (the N_(LF) ^(th) pupil partarea) of the image forming optical system are generated.

Here, shading due to pupil shifts of the first viewpoint image and thesecond viewpoint image (first to N_(LF) ^(th) viewpoint images) will bedescribed. In FIGS. 8A to 8C, a relationship of the first pupil partarea 401 in which the first photoelectric conversion unit 301 receiveslight at a peripheral image height of the imaging element, the secondpupil part area 402 in which the second photoelectric conversion unit302 receives light, and the exit pupil 410 of the image forming opticalsystem is illustrated. The same parts as those of FIGS. 4A and 4B aredenoted by the same reference signs. The first photoelectric conversionunit 301 and the second photoelectric conversion unit 302 (first toN_(LF) ^(th) photoelectric conversion units) correspond to the firstsub-pixel 201 and the second sub-pixel 202 (first to N_(LF) ^(th)sub-pixels), respectively.

FIG. 8A illustrates the case in which an exit pupil distance Dl of theimage forming optical system is the same as a set pupil distance Ds ofthe imaging element. In this case, the exit pupil 410 of the imageforming optical system is generally equally divided by the first pupilpart area 401 and the second pupil part area 402. On the other hand, ifthe exit pupil distance Dl of the image forming optical systemillustrated in FIG. 8B is shorter than the set pupil distance Ds of theimaging element, a pupil shift between the exit pupil of the imageforming optical system and the incident pupil of the imaging elementoccurs at the peripheral image height of the imaging element and theexit pupil 410 of the image forming system is unequally divided.Likewise, if the exit pupil distance Dl of the image forming opticalsystem illustrated in FIG. 8C is longer than the set pupil distance Dsof the imaging element, a pupil shift between the exit pupil of theimage forming optical system and the incident pupil of the imagingelement occurs at the peripheral image height of the imaging element andthe exit pupil 410 of the image forming system is unequally divided.When the pupil division is unequal at the peripheral image height,intensities of the first viewpoint image and the second viewpoint imageare unequal, and shading in which the intensity of one of the firstviewpoint image and the second viewpoint image increases and theintensity of the other decreases occurs for every RGB.

When necessary, to improve the shading of each viewpoint image, ashading correction process (an optical correction process) may beperformed on each of the first viewpoint image and the second viewpointimage (the first to N_(LF) ^(th) viewpoint images) for every RGB. Inaddition, when necessary, a defect correction process, a saturationprocess, de-mosaic processing, or the like may be performed.

In step S1 of FIG. 7, a process of generating a captured image (asynthesized image) according to a pupil area obtained by synthesizingdifferent pupil part areas of the image forming optical system isperformed. A captured image I(j, i) that is i^(th) in the columndirection and is j^(th) in the row direction is generated by Formula(2).

$\begin{matrix}{{I\left( {j,i} \right)} = {{\sum\limits_{k = 1}^{N_{LF}}{I_{k}\left( {j,i} \right)}} = {\sum\limits_{j_{S} = 1}^{N_{y}}{\sum\limits_{i_{S} = 1}^{N_{x}}{{{LF}\left( {{{N_{y}\left( {j - 1} \right)} + j_{S}},{{N_{x}\left( {i - 1} \right)} + i_{S}}} \right)}.}}}}} & (2)\end{matrix}$

In the present embodiment, an example of two divisions in the x-axisdirection for N_(x)=2, N_(y)=1, and N_(LF)=2 is shown. A process ofsynthesizing both signals of the first sub-pixel 201 and the secondsub-pixel 202 obtained by dividing each pixel into two sub-pixels in thex-axis direction from an input image corresponding to a pixel arrayillustrated in FIG. 3 and generating a captured image which is an RGBsignal of the Bayer array having a resolution of the number of pixels Nis performed. When necessary, a shading correction process, a defectcorrection process, a saturation process, de-mosaic processing, and thelike may be performed. FIG. 9 illustrates an example of a captured imagesubjected to the de-mosaic processing of the present embodiment. A dollthat is a main object is arranged in the center of FIG. 9 and a plate ofa fine checkered pattern sloping from the front side to the back side isarranged on the left side of FIG. 9.

In the present embodiment, as described above, a plurality of viewpointimages are generated for each different pupil part area from an inputimage acquired by the imaging element in which a plurality of pixelshaving a plurality of photoelectric conversion units for receiving lightbeams passing through different pupil part areas of the image formingoptical system are arrayed. A captured image according to a pupil areaobtained by synthesizing the different pupil part areas is generated.The present invention is not limited thereto, however, and the inventioncan be applied as long as a plurality of viewpoint images and asynthesized image thereof can be acquired by the well-known technologyin the present embodiment and the other embodiments. For example, as inJapanese Patent Laid-Open No. 2011-22796, different cameras of aplurality of viewpoints may be integrated and the integrated cameras maybe configured to be regarded as the imaging element 110. In addition,differently from the optical systems of FIGS. 2 and 3, an image of alight beam from the photographing optical system may be formed on amicrolens array, and the imaging element may be configured to beprovided on the image formation plane so that a physical object planeand the imaging element are in a conjugate relationship. Further, thelight beam from the photographing optical system may be subjected toimage reformation on the microlens array (this is referred to as imagereformation because image formation is performed in a state in which alight beam subjected to image formation once is diffused) and theimaging element may be configured to be provided on the image formationplane. Also, a method of inserting a mask having a suitable pattern (again modulation element) into an optical path of the photographingoptical system can be used.

Contrast Distribution

Next, a process of calculating the contrast distribution to be used inthe sharpness/unsharpness control will be described. In step S2 of FIG.7, the contrast distribution is generated by extracting a high-frequencyband component of a spatial frequency for every area from each of thecaptured image (the synthesized image) of the present embodiment and theplurality of viewpoint images. The contrast distribution of the presentembodiment is adjusted according to a difference between viewpointimages.

In step S2 of FIG. 7, an imaging luminance signal Y is first generatedaccording to Formula (3A) by causing the centers of gravity of thecolors RGB to match one another for every position (j, i) from acaptured image I(j, i) that is an RGB signal of the Bayer array.Likewise, a k^(th) viewpoint luminance signal Y_(k) is generatedaccording to Formula (3B) from a k^(th) viewpoint image I_(k) (k=1 toN_(LF)) that is an RGB signal of the Bayer array.

$\begin{matrix}{{{Y\left( {j,i} \right)} = {\begin{pmatrix}{I\left( {{j - 1},{i - 1}} \right)} & {I\left( {{j - 1},i} \right)} & {I\left( {{j - 1},{i + 1}} \right)} \\{I\left( {j,{i - 1}} \right)} & {I\left( {j,i} \right)} & {I\left( {j,{i + 1}} \right)} \\{I\left( {{j + 1},{i - 1}} \right)} & {I\left( {{j + 1},i} \right)} & {I\left( {{j + 1},{i + 1}} \right)}\end{pmatrix}\begin{pmatrix}\frac{1}{16} & \frac{2}{16} & \frac{1}{16} \\\frac{2}{16} & \frac{4}{16} & \frac{2}{16} \\\frac{1}{16} & \frac{2}{16} & \frac{1}{16}\end{pmatrix}}},} & \left( {3A} \right) \\{{Y_{k}\left( {j,i} \right)} = {\begin{pmatrix}{I_{k}\left( {{j - 1},{i - 1}} \right)} & {I_{k}\left( {{j - 1},i} \right)} & {I_{k}\left( {{j - 1},{i + 1}} \right)} \\{I_{k}\left( {j,{i - 1}} \right)} & {I_{k}\left( {j,i} \right)} & {I_{k}\left( {j,{i + 1}} \right)} \\{I_{k}\left( {{j + 1},{i - 1}} \right)} & {I_{k}\left( {{j + 1},i} \right)} & {I_{k}\left( {{j + 1},{i + 1}} \right)}\end{pmatrix}{\begin{pmatrix}\frac{1}{16} & \frac{2}{16} & \frac{1}{16} \\\frac{2}{16} & \frac{4}{16} & \frac{2}{16} \\\frac{1}{16} & \frac{2}{16} & \frac{1}{16}\end{pmatrix}.}}} & \left( {3B} \right)\end{matrix}$

Next, in step S2, using a two-dimensional band-pass filter forextracting a high-frequency component of a spatial frequency, an imaginghigh-frequency signal dy(j, i) is generated according to Formula (4A)from the imaging luminance signal Y(j, i). The two-dimensional band-passfilter is designated as {F_(BPF) (j_(BPF),i_(BPF))|−n_(BPF)≤j_(BPF)≤n_(BPF), −m_(BPF)≤i_(BPF)≤m_(BPF)}. Likewise,a k^(th) viewpoint high-frequency signal dY_(k) is generated accordingto Formula (4B) from the k^(th) viewpoint luminance signal Y_(k)(j, (k=1to N_(LF)).

$\begin{matrix}{{{{dY}\left( {j,i} \right)} = {{\sum\limits_{j_{BPF} = {- n_{BPF}}}^{n_{BPF}}{\sum\limits_{i_{BPF} = {- m_{BPF}}}^{m_{BPF}}{{F_{BPF}\left( {j_{BPF},i_{BPF}} \right)} \times {Y\left( {{j + j_{BPF}},{i + i_{BPF}}} \right)}}}}}},} & \left( {4A} \right) \\{{{dY}_{k}\left( {j,i} \right)} = {{{\sum\limits_{j_{BPF} = {- n_{BPF}}}^{n_{BPF}}{\sum\limits_{i_{BPF} = {- m_{BPF}}}^{m_{BPF}}{{F_{BPF}\left( {j_{BPF},i_{BPF}} \right)} \times {Y_{k}\left( {{j + j_{BPF}},{i + i_{BPF}}} \right)}}}}}.}} & \left( {4B} \right)\end{matrix}$

In the present embodiment, an example of two divisions in the x-axisdirection for N_(x)=2, N_(y)=1, and N_(LF)=2 is shown. An example inwhich the two-dimensional band-pass filter is configured by a directproduct of a one-dimensional filter Fx(i_(BPF)) of the x-axis direction(a pupil division direction) and a one-dimensional filter Fy(j_(BPF)) ofthe y-axis direction (a direction orthogonal to the pupil divisiondirection) is shown. That is, the two-dimensional band-pass filter isdesignated as F_(BPF)(j_(BPF), i_(BPF))=Fy(j_(BPF))×Fx(i_(BPF)). It ispossible to use a one-dimensional band-pass filter such as, for example,0.5×[1, 2, 0, −2, −1]+1.5×[1, 0, −2, 0, 1], to extract a high-frequencycomponent of a spatial frequency of the x-axis direction in theone-dimensional filter Fx(i_(BPF)) of the x-axis direction.

Here, a mixed filter obtained by combining a primary differential filter[1, 2, 0, −2, −1] and a secondary differential filter [1, 0, −2, 0, 1]is used. In general, when a differential filtering process is performed,0 point is in a portion at which there is change from the positive signto the negative sign in a signal after the filtering process. Thus, ajoint may be generated in an area including a high-frequency componentof the spatial frequency through combination with the calculation of anabsolute value. A position at which the joint is generated differsaccording to an order of differentiation of the differential filter.Consequently, in the present embodiment, the generation of the joint issuppressed using the mixed filter in which the primary differentialfilter and the secondary differential filter (generally, differentialfilters having different orders) are combined.

When necessary, a primary differential filter, such as [1, 2, 0, −2,−1], a secondary differential filter, such as [1, 0, −2, 0, 1], ahigh-order differential filter, or a more general primary band-passfilter may be used. It is possible to use a high-frequency cut(low-pass) filter such as, for example, [1, 1, 1, 1, 1] or [1, 4, 6, 4,1], to suppress high-frequency noise of the y-axis direction in theprimary filter Fy (J_(BPF)) of the y-axis direction orthogonal to thepupil division direction. When necessary, a low-pass filtering processof extracting a high-frequency component of a spatial frequency may beperformed in either the x-axis direction or the y-axis direction.Although an example of a two-dimensional band-pass filter configured bya direct product of two one-dimensional filters is shown in the presentembodiment, the present invention is not limited thereto, and a generaltwo-dimensional band-pass filter can be used.

Next, in step S2 of FIG. 7, a normalized imaging high-frequency signaldZ(j, i) obtained by normalizing the imaging high-frequency signal dY(j,i) by an imaging luminance signal Y(j, i) using Y₀>0 is generatedaccording to Formula (5A). Likewise, a normalized k^(th) viewpointhigh-frequency signal dZ_(k)(j, i) obtained by normalizing a k^(th)viewpoint high-frequency signal dY_(k)(j, i) (k=1 to N_(LF)) accordingto a k^(th) viewpoint luminance signal Y_(k)(j, i) is generatedaccording to Formula (5B). A determination of a maximum value of thehigh-frequency signal and Y₀ in the denominator is to prevent divisionby 0. When necessary, before the normalization in Formulas (5A) and(5B), a high-frequency cut (low-pass) filtering process is performed onthe imaging luminance signal y(j, i) and the k^(th) viewpoint luminancesignal Y_(k)(j, i) and high-frequency noise may be suppressed.dZ(j,i)=dY(j,i)/max(Y(j,i),Y ₀),  (5A)dZ _(k)(j,i)=dY _(k)(j,i)/max(Y _(k)(j,i),Y ₀).  (5B)

Next, in step S2, an imaging contrast distribution C(j, i) is generatedaccording to Formula (6A) using a low luminance threshold value Y_(min),a maximum contrast threshold value C_(max), and an exponent γ. If animaging luminance signal Y(j, i) is less than a low luminance thresholdvalue Y_(min) in the first row of Formula (6A), a value of the imagingcontrast distribution C(j, i) is set to 0. If the normalized imaginghigh-frequency signal dZ(j, i) is greater than a maximum contrastthreshold value C_(max) in the third row of Formula (6A), the value ofthe imaging contrast distribution C(j, i) is set to 1. Otherwise, theimaging contrast distribution C(j, i) is set to a value obtained bynormalizing the normalized imaging high-frequency signal dZ(j, i) by themaximum contrast threshold value C_(max) and obtaining the normalizedimaging high-frequency signal dZ(j, i) raised to the power of γ in thesecond row of Formula (6A). Likewise, the k^(th) viewpoint contrastdistribution C_(k)(j, i) (k=1 to N_(LF)) is generated according toFormula (6B).

$\begin{matrix}{{C\left( {j,i} \right)} = \left\{ \begin{matrix}0 & {\left( {{Y\left( {j,i} \right)} < Y_{\min}} \right),} \\\left( {{{dZ}\left( {j,i} \right)}/C_{\max}} \right)^{\gamma} & {\left( {{{dZ}\left( {j,i} \right)} \leq C_{\max}} \right),} \\1 & {\left( {{{dZ}\left( {j,i} \right)} > C_{\max}} \right).}\end{matrix} \right.} & \left( {6A} \right) \\{{C_{k}\left( {j,i} \right)} = \left\{ \begin{matrix}0 & {\left( {{Y\left( {j,i} \right)} < Y_{\min}} \right),} \\\left( {{{dZ}_{k}\left( {j,i} \right)}/C_{\max}} \right)^{\gamma} & {\left( {{{dZ}_{k}\left( {j,i} \right)} \leq C_{\max}} \right),} \\1 & {\left( {{{dZ}_{k}\left( {j,i} \right)} > C_{\max}} \right).}\end{matrix} \right.} & \left( {6B} \right)\end{matrix}$

As described above, the imaging contrast distribution C(j, i) has avalue within a range of [0, 1] (a value greater than or equal to 0 andless than or equal to 1). The contrast is indicated as being decreasedwhen a value of C(j, i) is close to 0 and increased when the value ofC(j, i) is close to 1. To adjust the tone curve from 0 to 1 of theimaging contrast distribution C(j, i), a value of a ratio between thenormalized high-frequency signal and the maximum contrast thresholdvalue raised to the power of γ is calculated. In order to alleviate achange at a low contrast side and steepen a change at a high contrastside, the exponent γ is desirably greater than or equal to 1.5 and lessthan or equal to 2.5. When necessary, a composite function F(C(j, i))may serve as an image contrast distribution using a function F:[0.1]→[0.1] from a domain [0, 1] to a range [0, 1].

A distribution example of the imaging contrast distribution C(j, i) ofthe present embodiment is illustrated in FIG. 10. Also, a distributionexample of a first viewpoint contrast distribution C₁(j, i) isillustrated in FIG. 11, and a distribution example of a second viewpointcontrast distribution C₂(j, i) is illustrated in FIG. 12. In thedistribution examples illustrated in FIGS. 10 to 12, a high/low index ofthe contrast is indicated by a gray scale of a range of [0, 1] on theright side. A white part in the vicinity of 1 indicates an area in whichthe number of high-frequency components of the spatial frequency of thex-axis direction is large and the contrast is high. Also, a black partin the vicinity of 0 indicates an area in which the number ofhigh-frequency components of the spatial frequency of the x-axisdirection is small and the contrast is low.

A relationship between a parallax between a first viewpoint image and asecond viewpoint image as a plurality of viewpoint images in the presentembodiment and perspective conflict or occlusion will be described usingFIGS. 13A to 13C. In FIGS. 13A to 13C, an imaging element (notillustrated) of the present embodiment is arranged in an imaging plane600, and the exit pupil of the image forming optical system is dividedinto two pupil part areas, i.e., the pupil part area 401 and the pupilpart area 402.

FIG. 13A is an example in which photographing is performed bysuperimposing a blurred image Γ1+Γ2 of a front-side object q2 with afocus image p1 of an object q1 and perspective conflict occurs in acaptured image. In this example, a light beam passing through the pupilpart area 401 of the image forming optical system and a light beampassing through the pupil part area 402 are divided and illustrated inFIGS. 13B and 13C, respectively.

In FIG. 13B, the light beam from the object q1 passes through the pupilpart area 401 and image formation is performed in an image p1 in afocused state, a light beam from a front-side object q2 passes throughthe pupil part area 401 and spreads to a blurred image Γ1 in a defocusedstate, and light is received in a sub-pixel 201 of each pixel of theimaging element. A first viewpoint image is generated from a lightreception signal of the sub-pixel 201. In the first viewpoint image, theimage p1 of the object q1 and the blurred image Γ1 of the front-sideobject q2 are captured at different positions without overlapping eachother. FIG. 13B is an example in which perspective conflict or occlusiondoes not occur among a plurality of objects (the object q1 and theobject q2) in the first viewpoint image.

On the other hand, in FIG. 13C, the light beam from the object q1 passesthrough the pupil part area 402 and image formation is performed in animage p1 in the focused state. A light beam from the front-side objectq2 passes through the pupil part area 402 and spreads to a blurred imageΓ2 in the defocused state and light is received in the sub-pixel 202 ofeach pixel of the imaging element. A second viewpoint image is generatedfrom a light reception signal of the sub-pixel 202. In the secondviewpoint image, the image p1 of the object q1 and the blurred image Γ2of the front-side object q2 overlap and are captured. FIG. 13C is anexample in which perspective conflict or occlusion occurs among aplurality of objects (the object q1 and the object q2) in the secondviewpoint image.

In the example of FIGS. 13A to 13C, a state in which perspectiveconflict or occlusion occurs in the first viewpoint image and the secondviewpoint image constituting the captured image in the vicinity of anarea in which perspective conflict or occlusion occurs in the capturedimage is different. That is, this indicates that a possibility of alarge difference between the first viewpoint image and the secondviewpoint image is high. Therefore, it is possible to estimate an areahaving a high possibility of occurrence of perspective conflict orocclusion by detecting an area of a large difference among the pluralityof viewpoint images.

An example of a difference amount distribution C₁(j, i)−C₂(j, i) betweena first viewpoint contrast distribution C₁(j, i) and a second viewpointcontrast distribution C₂(j, i) of the present embodiment is illustratedin FIG. 14. In the distribution example illustrated in FIG. 14, an indexof a magnitude for a difference between the contrast of the firstviewpoint image and the contrast of the second viewpoint image isindicated by a gray scale indication of a range of [−1, 1] on the rightside. This contrast difference corresponds to a difference amountbetween the first viewpoint contrast distribution and the secondviewpoint contrast distribution. A black portion close to 0 indicates anarea in which the contrast difference between the first viewpoint imageand the second viewpoint image is small. On the other hand, a whiteportion close to ±1 indicates an area in which the contrast differencebetween the first viewpoint image and the second viewpoint image islarge.

In FIG. 14, as the area in which the contrast difference between thefirst viewpoint image and the second viewpoint image is large, an areain which perspective conflict or occlusion occurs in the body of themain object (the doll) and the plate of the checkered pattern isdetected in the bottom center. Also, in addition to the area in whichthe perspective conflict or the occlusion occurs, an area in which ahigh-frequency band component of the spatial frequency is significantlychanged is detected in the first viewpoint image and the secondviewpoint image. For example, as in the object edge portion of thedefocused state, an area in which the high-frequency band component ofthe spatial frequency is significantly changed is detected in the firstviewpoint image and the second viewpoint image, such as an area in whichthe image shift amount is large in a state in which a high contrast ismaintained. In these detection areas, an object image having a largedifference in the spatial frequency component is captured in the firstviewpoint image and the second viewpoint image. Thus, in a capturedimage obtained by integrating the first viewpoint image and the secondviewpoint image, their detection areas are areas in which a plurality ofobject images having a large difference in the spatial frequencycomponent are mixed.

When image processing, such as sharpening or smoothing, is stronglyperformed in the area in which a plurality of object images havingdifferent spatial frequency components are mixed, image quality may bedegraded. Therefore, in the present embodiment, the area in which aplurality of object images having different spatial frequency componentsare mixed is detected using an absolute value |C₁(j, i)−C₂(j, i)| of thedifference amount distribution between the first viewpoint contrastdistribution C₁(j, i) and the second viewpoint contrast distributionC₂(j, i). The image processing, such as sharpening or smoothing, issuppressed and is performed in the detected mixed area. Thereby, it ispossible to perform image processing, such as sharpening or smoothing,while maintaining good image quality.

In the present embodiment, next, the contrast difference amountdistribution is generated to detect the area in which the plurality ofobject images having different spatial frequency components are mixed instep S2 of FIG. 7. In detail, the contrast difference amountdistribution C_(DIFF)(j, i) is generated according to Formula (7A) fromthe first viewpoint contrast distribution C₁(j, i) and the secondviewpoint contrast distribution C₂(j, i). Next, according to Formula(7B), an arithmetic process of multiplying the imaging contrastdistribution C(j, i) by the contrast difference amount distributionC_(DIFF)(j, i) is performed. Thereby, a contrast distribution M_(CON)(j,i) in which a value of the area, in which the plurality of object imageshaving the different spatial frequency components are mixed, issuppressed to be close to 0, is generated.C _(DIFF)(j,i)=1−|C ₁(j,i)−C ₂(j,i)|,  (7A)M _(CON)(j,i)=C(j,i)×C _(DIFF)(j,i).  (7B)

The contrast difference amount distribution C_(DIFF)(j, i) is adistribution of a range of [0, 1] and is close to a value of 0 in anarea in which the contrast difference between viewpoint images is largeand mixing of object images having different spatial frequencycomponents is great. Also, C_(DIFF)(j, i) is a distribution close to avalue of 1 in an area in which the contrast difference between theviewpoint images is small and mixing of object images having differentspatial frequency components is small. A contrast distributionM_(CON)(j, i) is a distribution in which the contrast difference amountdistribution C_(DIFF)(j, i) is combined with the imaging contrastdistribution C(j, i). Consequently, the distribution is a distributionin which a value of the area, in which the plurality of object imageshaving the different spatial frequency components are mixed, issuppressed to be close to 0.

A distribution example of the contrast distribution M_(CON)(j, i) of thepresent embodiment is illustrated in FIG. 15. In the distributionexample illustrated in FIG. 15, a high/low index of the contrast isindicated by a gray scale indication of a range of [0, 1] on the rightside. A white part in the vicinity of 1 indicates an area in which thenumber of high-frequency components of the spatial frequency of thex-axis direction is large and the contrast is high. Also, a black partin the vicinity of 0 indicates an area in which the number ofhigh-frequency components of the spatial frequency of the x-axisdirection is small and the contrast is low. The contrast value issuppressed in an area in which an absolute value |C₁(j, i)−C₂(j, i)| ofthe difference amount distribution between the first viewpoint contrastdistribution C₁(j, i) and the second viewpoint contrast distributionC₂(j, i) is large with respect to the imaging contrast distribution C(j,i) illustrated in FIG. 10.

In the present embodiment, a monotonically decreasing linear function isused for the absolute value |C₁(j, i)−C₂(j, i)| of the difference amountdistribution between the first viewpoint contrast distribution and thesecond viewpoint contrast distribution as the contrast difference amountdistribution C_(DIFF)(j, i). The present invention is not limitedthereto, however, and a more general function may be used whennecessary.

As described above, in the present embodiment, the contrast distributionM_(CON)(j, i) is generated as a composite contrast distributionaccording to a contrast difference for each viewpoint image from thecaptured image and the plurality of viewpoint images. In the contrastdistribution of the present embodiment, an area in which a differencebetween contrasts of each viewpoint image is small is greater than anarea in which a difference between contrasts is large. Also, in thecontrast distribution of the present embodiment, an area in which thenumber of spatial frequency components of the captured image in apredetermined spatial frequency band is large is greater than an area inwhich the number of spatial frequency components is small. Also, in thecontrast distribution of the present embodiment, an area in which theluminance of the captured image is high is greater than an area in whichthe luminance of the captured image is low.

In second and subsequent processes, for example, a process of recordingdistribution data is executed to omit the generation of the contrastdistribution M_(CON)(j, i) and shorten a processing time. That is, aprocess of recording the generated contrast distribution M_(CON)(j, i)in a recording medium, such as a flash memory, in association withrecorded image data is performed and distribution data is referred towhen necessary.

Image Shift Amount Distribution

In step S3 of FIG. 7, an image shift amount distribution is generated onthe basis of a correlation between two viewpoint images (a degree ofsignal matching) from the first viewpoint image and the second viewpointimage (a plurality of viewpoint images) at each position (j, i) at whicha value of the contrast distribution M_(CON)(j, i) is greater than orequal to a predetermined value. Also, the present invention is notlimited thereto, and the image shift amount distribution may begenerated on the basis of each viewpoint image regardless of a value ofthe contrast distribution M_(CON)(j, i).

In step S3, a one-dimensional band-pass filtering process is firstperformed on a first viewpoint luminance signal Y₁ generated accordingto Formula (3B) from a first viewpoint image I₁ that is an RGB signal ofa Bayer array in a pupil division direction (a column direction).Thereby, a first focus detection signal dYA is generated. Also, aone-dimensional band-pass filtering process is performed on a secondviewpoint luminance signal Y₂ generated according to Formula (3B) from asecond viewpoint image I₂ in the pupil division direction (the columndirection). Thereby, a second focus detection signal dYB is generated.As the one-dimensional band-pass filter, for example, a one-dimensionaldifferential filter [1, 5, 8, 8, 8, 8, 5, 1, −1, −5, −8, −8, −8, −8, −5,−1] or the like can be used. When necessary, a pass band of theone-dimensional band-pass filter may be adjusted.

In step S3, a correlation amount is calculated at each position (j, i)at which a value of the contrast distribution M_(CON)(j, i) is greaterthan or equal to a predetermined value (for example 0.2). A process ofrelatively shifting the first focus detection signal dYA and the secondfocus detection signal dYB in the pupil division direction (the columndirection) and calculating a correlation amount indicating a degree ofsignal matching is executed. An image shift amount distributionM_(DIS)(j, i) is generated on the basis of a correlation amount. On theother hand, each position (j, i) at which a value of the contrastdistribution M_(CON)(j, i) is less than the predetermined value (forexample 0.2) is excluded from the calculation of the image shift amount.It is possible to increase the precision of detection of the image shiftamount and increase the speed of processing by limiting the detection ofthe image shift amount to an area in which perspective conflict orocclusion does not occur with a high contrast.

A first focus detection signal that is j₂ ^(th)(−n₂≤j₂≤n₂) in the rowdirection around the position (j, i) and is i₂ ^(th)(−m₂≤i₂≤m₂) in thecolumn direction that is a pupil division direction is denoted bydYA(j+j₂, i+i₂), and a second focus detection signal is denoted bydYB(j+j₂, i+i₂). Using a shift amount as s (−n_(s)≤s≤n_(s)), thecorrelation amount at each position (j, i) is denoted by COR_(even)(i,j, s) and COR_(odd)(i, j, s). The correlation amount COR_(even)(i, j, s)is calculated according to Formula (8A) and the correlation amountCOR_(odd)(i, j, s) is calculated according to Formula (8B).

$\begin{matrix}{{{COR}_{even}\left( {j,i,s} \right)} = {\sum\limits_{j_{2} = {- n_{2}}}^{n_{2}}{\sum\limits_{i_{2} = {- m_{2}}}^{m_{2}}{{{{dYA}\left( {{j + j_{2}},{i + i_{2} + s}} \right)} - {{dYB}\left( {{j + j_{2}},{i + i_{2} - s}} \right)}}}}}} & \left( {8A} \right) \\{{{COR}_{odd}\left( {j,i,s} \right)} = {\sum\limits_{j_{2} = {- n_{2}}}^{n_{2}}{\sum\limits_{i_{2} = {- m_{2}}}^{m_{2}}{{{{dYA}\left( {{j + j_{2}},{i + i_{2} + s}} \right)} - {{dYB}\left( {{j + j_{2}},{i + i_{2} - 1 - s}} \right)}}}}}} & \left( {8B} \right)\end{matrix}$

The correlation amount COR_(odd)(j, i, s) is a correlation amountobtained by shifting shift amounts of the first focus detection signaldYA and the second focus detection signal dYB by (half phase—1) withrespect to the correlation amount COR_(even)(i, j, s).

An average value is calculated by calculating a shift amount of a realvalue in which the correlation amount is a minimum value according toeach sub-pixel calculation from the correlation amount COR_(even)(i, j,s) and the correlation amount COR_(odd)(i, j, s), and an image shiftamount distribution M_(DIS)(j, i) is generated. In an area in which avalue of the contrast distribution M_(CON)(j, i) is less than thepredetermined value (for example, 0. 2), the image shift amountdistribution M_(DIS)(j, i) is excluded from the calculation of the imageshift amount, for example, M_(DIS)(j, i)=0. When necessary, a valueother than 0 may be set.

A distribution example of an image shift amount distribution M_(DIS)(j,i) of the present embodiment is illustrated in FIG. 16. A gray scaleindication of a range of [−6, 6] is illustrated on the right side. InFIG. 16, an image shift amount between the first viewpoint image and thesecond viewpoint image is indicated in units of pixels by a gray scaleindication in an area in which an image shift amount is calculated whenthe value of the contrast distribution M_(CON)(j, i) is greater than orequal to a predetermined value 0.2. A portion of a black side of minus(−) indicates a front focus state and a portion in the vicinity of 0indicates an area near the focus. A portion of a white side of plus (+)indicates a rear focus state. In addition, in the display of thedistribution example of FIG. 16, a value of the contrast distributionM_(CON)(j, i) is excluded from the calculation of the image shift amountwhen the value is less than the predetermined value 0.2. That is, anarea in which M_(DIS)(j, i)=0 is set is indicated in a black color.

As described above, in the present embodiment, the image shift amountdistribution M_(DIS)(j, i) is generated from a plurality of viewpointimages. In second and subsequent processes, for example, a process ofrecording the generated image shift amount distribution M_(DIS)(j, i) isexecuted to omit the generation of the image shift amount distributionM_(DIS)(j, i) and shorten a processing time. That is, the image shiftamount distribution data is recorded in a recording medium, such as aflash memory, in association with recorded image data. When necessary,conversion into the defocus amount distribution indicating adistribution of a defocus amount of the object within the viewpointimage may be performed by multiplying the image shift amountdistribution M_(DIS)(j, i) by a conversion coefficient according to aposition (j, i), an aperture value of an imaging lens (an image formingoptical system), an exit pupil distance, or the like.

Image Shift Different Amount Distribution

In step S4 of FIG. 7, a process of generating the image shift differenceamount distribution M_(DIFF)(j, i) from the image shift amountdistribution M_(DIS)(j, i) and the predetermined image shift amount isexecuted. In step S4, an image shift amount desired to be corrected bythe refocus process of the present embodiment is first set as thepredetermined image shift amount p. For example, in the example of theimage shift amount distribution M_(DIS) of FIG. 16, an image shiftamount in the area near the eye is about 2.5. In the refocus process, ifan image shift amount in an area near an eye of the main object (thedoll) is desired to be finely corrected to generally 0, a predeterminedimage shift amount is set to p=2.5.

Next, in step S4, the image shift difference amount distributionM_(DIFF)(j, i) is calculated according to Formula (9) from the imageshift amount distribution M_(DIS)(j, i), the predetermined image shiftamount p, and the contrast distribution M_(CON)(j, i) using σ_(p)>0.

$\begin{matrix}{{M_{DIFF}\left( {j,i} \right)} = {\left( {1 - \frac{{{M_{DIS}\left( {j,i} \right)} - p}}{\sigma_{p}}} \right) \times {M_{CON}\left( {j,i} \right)}}} & (9)\end{matrix}$

The image shift difference amount distribution M_(DIFF)(j, i) is adistribution in which linear functions, in which there is monotonicdecrease of an absolute value |M_(DIS)(j, i)−p| of a difference betweenthe image shift amount distribution M_(DIS)(j, i) and the predeterminedimage shift amount p and the contrast distribution M_(CON)(j, i), arecombined. The image shift difference amount distribution M_(DIFF)(j, i)is positive for |M_(DIS)(j, i)−p|<σ_(p), is 0 for |M_(DIS)(j,i)−p|=σ_(p), and is negative for |M_(DIS)(j, i)−p|>σ_(p).

In an area in which the value of the contrast distribution M_(CON)(j, i)is less than the predetermined value (for example, 0.2) and is excludedfrom the calculation of the image shift amount, M_(DIFF)(j,i)=(1−|p|/σ_(p))×M_(CON)(j, i). When necessary, another value may beset.

A distribution example of an image shift difference amount distributionM_(DIFF)(j, i) of the present embodiment is illustrated in FIG. 17. Agray scale indication of a range of [−1, 1] on the right side is shown.In the area in which the image shift amount is calculated when the valueof the contrast distribution M_(CON) is greater than or equal to apredetermined value 0.2, an image shift difference amount is indicatedby the gray scale indication. A portion of a white side of a plus (+)sign indicates an area in which the absolute value |M_(DIS)(j, i)−p|between the image shift amount distribution M_(DIS)(j, i) and thepredetermined image shift amount p is small and the contrast is high. Aportion of a white side of a minus (−) sign indicates an area in whichthe absolute value |M_(DIS)(j, i)−p| between the image shift amountdistribution M_(DIS)(j, i) and the predetermined image shift amount p islarge and the contrast is high. Also, in the display of the distributionexample of FIG. 17, a value of the contrast distribution M_(CON)(j, i)is less than the predetermined value 0.2 and is excluded from thecalculation of the image shift amount. That is, an area in whichM_(DIFF)(j, i)=(1−|p|/σ_(p))×M_(CON)(j, i) is set is indicated in ablack color.

Corrected Viewpoint Image

In step S5 of FIG. 7, processes of first sharpening and first smoothingare performed on the first viewpoint image and the second viewpointimage (first to N_(LF) ^(th) viewpoint images) according to the imageshift difference amount distribution M_(DIFF)(j, i). A first correctedviewpoint image and a second corrected viewpoint image (first to N_(LF)^(th) corrected viewpoint images) are generated.

In the present embodiment, a (crosstalk correction, first sharpening)process of enlarging a difference between viewpoint images andsharpening a parallax in an area in which the image shift differenceamount distribution is greater than or equal to 0 (M_(DIFF)(j, i)≥0)with respect to the first viewpoint image and the second viewpoint image(a plurality of viewpoint images) is performed. On the other hand, in anarea in which the image shift difference amount distribution is lessthan 0 (M_(DIFF)(j, i)<0), a (crosstalk correction, first smoothing)process of smoothing the parallax by reducing the difference between theviewpoint images is performed. According to the above-described process,the first corrected viewpoint image and the second corrected viewpointimage (the plurality of corrected viewpoint images) are generated.

In step S5 of FIG. 7, the strength of the (crosstalk correction, firstsharpening) process of enlarging a difference between viewpoint imageswith respect to the first viewpoint image and the second viewpoint image(the plurality of viewpoint images) and sharpening a parallax is firstset. A first strength parameter for designating the strength of thepresent process is denoted by k_(ct) and k_(ct)≥0. Alternatively, thestrength parameter k_(ct)≥0 for designating the strength of the(crosstalk correction, first smoothing) process of smoothing theparallax by reducing the difference between the viewpoint images is setas the first strength parameter.

Next, in step S5, the first strength parameter distribution K_(ct)(j, i)is set according to Formula (10). The first strength parameterdistribution K_(ct)(j, i) is proportional to the image shift differenceamount distribution M_(DIFF)(j, i) using k_(ct) as a proportionalcoefficient.K _(ct)(j,i)=k _(ct) ×M _(DIFF)(j,i),  (10)

Next, in step S5, arithmetic processes of Formula (11A) and Formula(11B) are performed with respect to the first viewpoint image I₁(j, i)and the second viewpoint image I₂(j, i) (first to N_(LF) ^(th) viewpointimages). A first corrected viewpoint image MI₁(j, i) and a secondcorrected viewpoint image MI₂(j, i) (first to N_(LF) ^(th) correctedviewpoint images) are generated.

$\begin{matrix}{{\begin{pmatrix}{{MI}_{1}\left( {j,i} \right)} \\{{MI}_{2}\left( {j,i} \right)}\end{pmatrix} = {\begin{pmatrix}{1 + {K_{ct}\left( {j,i} \right)}} & {- {K_{ct}\left( {j,i} \right)}} \\{- {K_{ct}\left( {j,i} \right)}} & {1 + {K_{ct}\left( {j,i} \right)}}\end{pmatrix}\begin{pmatrix}{I_{1}\left( {j,i} \right)} \\{I_{2}\left( {j,i} \right)}\end{pmatrix}}},\left( {{K_{ct}\left( {j,i} \right)} \geq 0} \right),} & \left( {11A} \right) \\{{\begin{pmatrix}{{MI}_{1}\left( {j,i} \right)} \\{{MI}_{2}\left( {j,i} \right)}\end{pmatrix} = {\begin{pmatrix}\frac{1 - {K_{ct}\left( {j,i} \right)}}{1 - {2{K_{ct}\left( {j,i} \right)}}} & \frac{- {K_{ct}\left( {j,i} \right)}}{1 - {2{K_{ct}\left( {j,i} \right)}}} \\\frac{- {K_{ct}\left( {j,i} \right)}}{1 - {2{K_{ct}\left( {j,i} \right)}}} & \frac{1 - {K_{ct}\left( {j,i} \right)}}{1 - {2{K_{ct}\left( {j,i} \right)}}}\end{pmatrix}\begin{pmatrix}{I_{1}\left( {j,i} \right)} \\{I_{2}\left( {j,i} \right)}\end{pmatrix}}},{\left( {{K_{ct}\left( {j,i} \right)} < 0} \right).}} & \left( {11B} \right)\end{matrix}$

The process of Formula (11A) is a process of sharpening a parallax byenlarging a difference between the first viewpoint image and the secondviewpoint image in an area in which the first strength parameterdistribution (the image shift difference amount distribution) is greaterthan or equal to 0 (K_(ct)(j, i)=k_(ct)×M_(DIFF)(j, i)≥0). On the otherhand, the process of Formula (11B) is a process of smoothing theparallax by reducing a difference between the first viewpoint image andthe second viewpoint image in an area in which the first strengthparameter distribution (the image shift difference amount distribution)is less than 0 (K_(ct)(j, i)=k_(ct)×M_(DIFF)(j, i)<0).

FIG. 18 illustrates an example of a (crosstalk correction, firstsharpening) process of sharpening a parallax by enlarging the differencebetween the first viewpoint image and the second viewpoint image in agraph. The horizontal axis represents a pixel position and the verticalaxis represents a pixel value (a signal level). In FIG. 18, examples ofa first viewpoint image (pre-correction A) and a second viewpoint image(pre-correction B) before the crosstalk correction and the firstsharpening process are illustrated in a graph of a dotted line. Also,examples of a first corrected viewpoint image (post-correction A) and asecond corrected viewpoint image (post-correction B) after the crosstalkcorrection and the first sharpening process according to Formula (11A)are illustrated in a graph of a solid line. In the (crosstalkcorrection, first sharpening) process of sharpening the parallax byenlarging the difference between the viewpoint images, a portion inwhich the difference between the viewpoint images is large before theprocess is further enlarged, but a portion in which the differencebetween the viewpoint images is small is hardly changed. As describedabove, it can be seen that the parallax between the viewpoint images issharpened.

On the other hand, in the smoothing process (crosstalk correction, firstsmoothing) according to Formula (11B), a difference between the firstviewpoint image and the second viewpoint image (the plurality ofviewpoint images) is reduced and the parallax between the viewpointimages is smoothed.

In the present embodiment, as described above, image processing ofsharpening and smoothing according to the contrast distribution and theimage shift amount distribution is performed on a plurality of viewpointimages. The image processing according to the contrast distribution andthe image shift amount distribution may be a sharpening process, thesmoothing process, or a combined process thereof when necessary.

In the present embodiment, arithmetic processes of Formula (7A), Formula(7B), Formula (9), Formula (10), Formula (11A), and Formula (11B) areperformed. The image processing, such as sharpening or smoothing, oneach viewpoint image is more strongly performed in an area in which adifference between contrasts is small than in an area in which thedifference between the contrasts is large for each viewpoint image.Also, the image processing, such as sharpening or smoothing, on eachviewpoint image is more strongly performed in an area in which acontrast distribution is large than in an area in which the contrastdistribution is small.

In the present embodiment, according to Formula (9), Formula (10),Formula (11A), and Formula (11B), a sharpening process is performed inan area in which a difference from the predetermined shift amount(reference) of the image shift amount distribution is small and asmoothing process is performed in an area in which the difference islarge. According to Formula (9), Formula (10), and Formula (11A), thesharpening process is strongly performed in an area in which adifference from a predetermined shift amount of the image shift amountdistribution is small than in an area in which the difference is large.Also, according to Formula (9), Formula (10), and Formula (11B), thesmoothing process is strongly performed in an area in which a differencefrom a predetermined shift amount of the image shift amount distributionis large than in an area in which the difference is small.

Also, in the present embodiment, a process of sharpening a parallax byenlarging a difference among a plurality of viewpoint images for eachpixel of the plurality of viewpoint images or a process of smoothing theparallax by reducing the difference among the plurality of viewpointimages is performed according to Formula (11A) and Formula (11B) and aplurality of corrected viewpoint images are generated. The firstsharpening process of Formula (11A) and the second smoothing process ofFormula (11B) are arithmetic processes between the first viewpoint imageI₁(j, i) and the second viewpoint image I₂(j, i). A signal of the firstviewpoint image I₁(j, i) is an output signal of the first photoelectricconversion unit included in the pixel of each position (j, i), and asignal of the second viewpoint image I₂(j, i) is an output signal of thesecond photoelectric conversion unit included in the pixel of eachposition (j, i).

Weight Coefficient

In step S6 of FIG. 7, a weight coefficient for each of a first correctedviewpoint image and a second corrected viewpoint image (first to N_(LF)^(th) corrected viewpoint images) is set to finely correct a depth offield in a predetermined area.

In step S6, a predetermined area R=[j₁, j₂]×[i₁, i₂], in which the depthof field is desired to be re-corrected, and a boundary width σ of thepredetermined area are first set. According to Formula (12), a tablefunction T(j, i) according to the predetermined area R and the boundarywidth σ of the predetermined area is calculated according to Formula(12).

$\begin{matrix}{{T\left( {j,i} \right)} = {0.5*\left\lbrack {{\tan\; h\frac{\left( {j - j_{1}} \right)}{\sigma}} - {\tan\; h\frac{\left( {j - j_{2}} \right)}{\sigma}}} \right\rbrack \times 0.5*{\left\lbrack {{\tan\; h\frac{\left( {i - i_{1}} \right)}{\sigma}} - {\tan\; h\frac{\left( {i - i_{2}} \right)}{\sigma}}} \right\rbrack.}}} & (12)\end{matrix}$

In Formula (12), tan h denotes a hyperbolic tangent function. The tablefunction T(j, i) has 1 inside the predetermined area R and has 0 outsidethe predetermined area R. In the boundary width σ of the predeterminedarea R, a value generally continuously changes from 1 to 0. Whennecessary, the predetermined area is a circular shape or may be anyshape. Also, when necessary, a plurality of predetermined areas andboundary widths may be set.

Next, in step S6, a first weight coefficient W₁(j, i) of the firstcorrected viewpoint image MI₁(j, i) is calculated as a real coefficientw (−1≤w≤1) according to Formula (13A). Also, a second weight coefficientW₂(j, i) of the second corrected viewpoint image MI₂(j, i) is calculatedaccording to Formula (13B).W ₁(j,i)=1−wT(j,i),  (13A)W ₂(j,i)=1+wT(j,i).  (13B)

If the depth of field is corrected by increasing an addition ratio ofthe first corrected viewpoint image MI₁(j, i) in a predetermined area,the depth of field is set in a range of −1≤w<0. Also, if the depth offield is corrected by increasing an addition ratio of the secondcorrected viewpoint image MI₂(j, i), the depth of field is set in arange of 0<w≤1. When necessary, the depth of field may not be correctedby setting w=0 and W₁≡W₂≡1.

Refocusing in Shift Synthesis Process

In step S7 of FIG. 7, a process (shift synthesis process) of multiplyingeach of a first corrected viewpoint image and a second correctedviewpoint image (first to N_(LF) ^(th) corrected viewpoint images) by aweight coefficient to relatively shift each of a first correctedviewpoint image and a second corrected viewpoint image (the first toN_(LF) ^(th) corrected viewpoint images) in the pupil division direction(the x-axis direction), and adding the shifted images is performed. Anintermediate image that is a synthesized image from the plurality ofviewpoint images is generated.

FIG. 19 is an explanatory diagram illustrating an overview of refocusingin a shift synthesis process of the pupil division direction (the x-axisdirection) based on the first corrected viewpoint image MI₁(j, i) andthe second corrected viewpoint image MI₂(j, i) (the plurality ofcorrected viewpoint images). In FIG. 19, the down direction is definedas a positive direction of the x-axis by setting the x-axis in theup/down direction of the paper surface, the front side is defined as apositive direction of the y-axis by setting a direction perpendicular tothe paper surface as the y-axis, and the left direction is defined as apositive direction of the z-axis by setting the z-axis in the left/rightdirection of the paper surface. An imaging plane 600 of FIG. 19corresponds to the imaging plane 600 illustrated in FIGS. 13A to 13C.

In FIG. 19, the first corrected viewpoint image MI₁(j, i) and the secondcorrected viewpoint image MI₂(j, i) are schematically illustrated. Asignal of the first corrected viewpoint image MI₁(j, i) is a lightreception signal of a light beam incident on the first photoelectricconversion unit 301 of the position (j, i) at a main ray angle θ₁corresponding to a first pupil part area 401 of FIGS. 13A to 13C. Asignal of the second corrected viewpoint image MI₂(j, i) is a lightreception signal of a light beam incident on the second photoelectricconversion unit 302 of the position (j, i) at a main ray angle θ₂corresponding to a second pupil part area 402 of FIGS. 13A to 13C. Thefirst photoelectric conversion unit 301 and the second photoelectricconversion unit 302 (the first to N_(LF) ^(th) photoelectric conversionunits) correspond to the first sub-pixel 201 and the second sub-pixel202 (the first to N_(LF) ^(th) sub-pixels), respectively.

The first corrected viewpoint image MI₁(j, i) and the second correctedviewpoint image MI₂(j, i) (the plurality of corrected viewpoint images)have incident angle information as well as light intensity distributioninformation. It is possible, therefore, to generate a refocus image in avirtual image formation plane 610 in the following parallel movement andaddition processes. A first process is a process of moving the firstcorrected viewpoint image MI₁(j, i) to the virtual image formation plane610 along the main ray angle θ₁ in parallel, and moving the secondcorrected viewpoint image MI₂(j, i) to the virtual image formation plane610 along the main ray angle θ₂ in parallel and a second process is aprocess of adding the first corrected viewpoint image MI₁(j, i) and thesecond corrected viewpoint image MI₂(j, i) moved in parallel.

Moving the first corrected viewpoint image MI₁(j, i) to the virtualimage formation plane 610 along the main ray angle θ₁ in parallelcorresponds to a shift of −1 pixel in the column direction. Also, movingthe second corrected viewpoint image MI₂(j, i) to the virtual imageformation plane 610 along the main ray angle θ₂ in parallel correspondsto a shift of +1 pixel in the column direction. It is possible,therefore, to generate a refocus signal in the virtual image formationplane 610 by relatively shifting the first corrected viewpoint imageMI₁(j, i) and the second corrected viewpoint image MI₂(j, i) by +2pixels and associating and adding MI₁(j, i) and MI₂(j, i+2).

In step S7 of FIG. 7, a process of generating a shift synthesis imageI_(S)(j, i) according to Formula (14) from the first corrected viewpointimage MI₁(j, i) and the second corrected viewpoint image MI₂(j, i) (theplurality of corrected viewpoint images) is performed. That is, ashifted synthesized image I_(S)(j, i) that is a refocus image in avirtual image formation plane is generated. An even number closest to apredetermined image shift amount p is denoted by pe. Here, the evennumber pe closest to the predetermined image shift amount p iscalculated according to pe=2×ROUND(p/2) using ROUND as a function forrounding-off.I _(S)(j,i)=W ₁(j,i)×MI ₁(j,i)+W ₂(j,i)×MI ₂(j,i−pe).  (14)

In Formula (14), simultaneously with the shift addition, the firstcorrected viewpoint image MI₁(j, i) is multiplied by a first weightcoefficient distribution W₁(j, i) of Formula (13A), and the secondcorrected viewpoint image MI₂(j, i) is multiplied by a second weightcoefficient distribution W₂(j, i) of Formula (13B). Thereby, it ispossible to correct a depth of field in a predetermined area. Whennecessary, the depth of field may not be corrected using W₁≡W₂≡1. Ashift synthesis process is performed by multiplying each of theplurality of viewpoint images by a weight coefficient, and anintermediate image that is a synthesized image from the plurality ofviewpoint images is generated.

The shift synthesis process of the first corrected viewpoint imageMI₁(j, i) and the second corrected viewpoint image MI₂(j, i) (theplurality of corrected viewpoint images) is not limited to an evennumber shift or an addition process. When necessary, a real number shiftor a more general synthesis process may be used. Also, when necessary,the process of step S8 of FIG. 7, to be described below, may be omitted,and the shifted synthesized image I_(S)(j, i) generated by shifting andadding the plurality of corrected viewpoint images according to Formula(14) may be an output image.

In the present embodiment, the number of pixels of the shiftedsynthesized image I_(S)(j, i) generated according to Formula (14) ismaintained at the same number as the number of pixels N of the capturedimage. Thus, a termination process of enlarging a data length isperformed on a termination portion in the pupil division direction (thex-axis direction) of the second corrected viewpoint image MI₂(j, i) inadvance. If pe>0, a termination process is executed according to Formula(15A) for a termination column number i_(e)(i_(min)≤i_(e)≤i_(min)+pe−1)using a minimum column number as I_(min). If pe<0, a termination processis executed according to Formula (15B) for a termination column numberi_(e) (i_(max)+pe+1≤i_(e)≤i_(max)) using a maximum column number asI_(max). In the present embodiment, a process of extending an image sizeof the plurality of corrected viewpoint images is performed.MI ₁(j,i _(e))=MI ₂(j,i _(min) +pe+mod(i _(e) −i_(min),2)),(pe>0),  (15A)MI ₂(j,i _(e))=MI ₂(j,i _(max) +pe−mod(i _(e) −i_(max),2)),(pe<0),  (15B)Refocusable Range

The refocusable range in the shift synthesis process in the presentembodiment will be described with reference to a schematic diagram ofFIG. 20. An imaging element (not illustrated) is arranged in the imagingplane 600. As in the case of FIGS. 13A to 13C, an exit pupil of theimage forming optical system is divided into a first pupil part area 401and a second pupil part area 402 according to (2×1) division.

When an allowable confusion circle diameter is denoted by δ and anaperture value of the image forming optical system is denoted by F, adepth of field in the aperture value F is ±F×δ. On the other hand, aneffective aperture value F₀₁ (or F₀₂) of the pupil division direction(the x-axis direction) of the pupil part area 401 (or 502) narrowed tobe divided into N_(x)×N_(y) (for example, 2×1) becomes F₀₁=N_(x)×F (orF₀₂=N_(x)×F) and it is darkening. The effective depth of field for eachfirst corrected viewpoint image (or second corrected viewpoint image) isincreased by a factor of N_(x) in ±N_(x)×F×δ and the focus range isincreased by a factor of N_(x). Within the range of the effective depthof field “±N_(x)×F×δ,” an object image focused for each first correctedviewpoint image (or second corrected viewpoint image) is acquired.Consequently, it is possible to refocus the focus position afterphotographing in a process of shifting the first corrected viewpointimage (or the second corrected viewpoint image) along the main ray angleθ₁ (or θ₂), illustrated in FIG. 19, in parallel, and adding the shiftedfirst corrected viewpoint image (or second corrected viewpoint image).

A defocus amount d from the imaging plane 600, in which the focusposition after the photographing can be refocused, is limited. Therefocusable range of the defocus amount d is generally a range ofFormula (16). The allowable confusion circle diameter δ is defined byδ=2·ΔX (a reciprocal of a Nyquist frequency 1/(2·ΔX) of a pixel cycleΔX) or the like.|d|≤N _(x) ×F×δ  (16)

As illustrated in the pupil distribution example of FIG. 5B, however, inthe pupil division by microlenses with a diameter of several micrometersand the photoelectric conversion unit divided into a plurality of partsformed for each pixel unit, the gradual pupil division is performedbecause of a diffraction blur due to the wave nature of light. Thus,even when the degree of focus depth in the pupil division direction (thex-axis direction) of the first viewpoint image and the second viewpointimage (the plurality of viewpoint images) is not sufficiently deep, andthe refocus image is generated using the first viewpoint image and thesecond viewpoint image (the plurality of viewpoint images), a refocuseffect may not be sufficiently obtained.

Therefore, in the present embodiment, the following process is performedon the first viewpoint image and the second viewpoint image (theplurality of viewpoint images) in refocusing in the shift synthesisprocess. A process of sharpening a parallax by enlarging a differencebetween the first viewpoint image and the second viewpoint imageaccording to Formula (11A) for each pixel for which a first intensityparameter distribution (an image shift difference amount distribution)is greater than or equal to 0 (K_(ct)(j, i)=k_(ct)×M_(DIFF)(j, i)≥0) isperformed. Therefore, the first corrected viewpoint image and the secondcorrected viewpoint image (the plurality of corrected viewpoint images)are generated. Thereby, it is possible to increase the effectiveaperture value F in the pupil division direction of the first correctedviewpoint image and the second corrected viewpoint image, to correct adegree of focus depth to be deeper, and to improve a refocus effect.

Hereinafter, an effect of a (crosstalk correction, first sharpening)process of sharpening the first viewpoint image and the second viewpointimage (the plurality of viewpoint images) in refocusing in the shiftsynthesis process will be described with reference to FIGS. 21A and 21B.FIG. 21A illustrates an example of a refocus image in the shiftsynthesis process of the first viewpoint image and the second viewpointimage before sharpening (crosstalk correction, first sharpening) in theconventional example.

For example, because the pupil division is gradual, the degree of focusdepth in the pupil division direction (the x-axis direction) of thefirst viewpoint image and the second viewpoint image (the plurality ofviewpoint images) is not sufficiently deep. Refocusing in the shiftsynthesis process is performed on a captured image of a rear focus statein which a focus is aligned behind a right eye of the main object (thedoll), but a sufficient refocus effect is not obtained while the righteye, the eyelashes, the hair, or the like of the main object (the doll)is in a small blur state.

On the other hand, FIG. 21B illustrates an example of a refocus image ina shift synthesis process on the first corrected viewpoint image and thesecond corrected viewpoint image after sharpening (crosstalk correction,first sharpening) in the present embodiment. For example, in the processof sharpening a parallax by enlarging the difference between the firstviewpoint image and the second viewpoint image, an effective aperturevalue F of the pupil division direction (the x-axis direction) of thefirst corrected viewpoint image and the second corrected viewpoint image(the plurality of corrected viewpoint images) increases and the degreeof focus depth is corrected to be deeper. According to refocusing in theshift synthesis process, after photographing, a focus position alignedin the right eye, the eyelashes, the hair, or the like of the mainobject (the doll) is re-corrected and the refocus effect is improved.

Also, if the number of pupil divisions is small and the number ofviewpoint images is small as in the present embodiment of two divisionsin the pupil division direction (the x-axis direction) for N_(x)=2,N_(y)=1, and N_(LF)=2, the following problems may occur. That is, in anarea in which a blur amount (an image shift amount) is increased in therefocusing in the shift synthesis process, a boundary of the object inwhich an artificial two-line blur is caused is doubly formed, and imagequality may be degraded.

Therefore, in the present embodiment, the following process is performedon the first viewpoint image and the second viewpoint image (theplurality of viewpoint images) in the refocusing in the shift synthesisprocess. A process (a first smoothing process) of smoothing a parallaxby reducing the difference between the first viewpoint image and thesecond viewpoint image according to Formula (11B) for each pixel forwhich a first intensity parameter distribution (an image shiftdifference amount distribution) is less than 0 (K_(ct)(j,i)=k_(ct)×M_(DIFF)(j, i)<0) is performed. Therefore, the first correctedviewpoint image and the second corrected viewpoint image (the pluralityof corrected viewpoint images) are generated. Thereby, in an area inwhich a blur amount (an image shift amount) is increased, it is possibleto perform refocusing in the synthesis process while suppressing thegeneration of the artificial two-line blur and maintaining good imagequality.

Sharpness/Unsharpness Control

Processes of second sharpening and second smoothing are executed in stepS8 of FIG. 7. A process of sharpening and smoothing according to theimage shift difference amount distribution M_(DIFF)(j, i) is performedon a shifted synthesized image (an intermediate image) generated fromthe first corrected viewpoint image and the second corrected viewpointimage (the first to N_(LF) ^(th) corrected viewpoint images). Accordingto this process, it is possible to generate an output image on which thesharpness/unsharpness control is performed to adaptively control an areain which the degree of sharpness is high and an area in which a degreeof blur is high after photographing.

In the present embodiment, the second sharpening process is performed onthe shifted synthesized image I_(S)(j, i) in an area in which the imageshift difference amount distribution is greater than or equal to 0(M_(DIFF)(j, i)≥0). On the other hand, the second smoothing process isperformed in an area in which the image shift difference amountdistribution is less than 0 (M_(DIFF)(j, i)<0). Therefore, an outputimage is generated.

In step S8 of FIG. 7, first, a second strength parameter k_(USM)≥0 forsetting the strength of the second sharpening process or the secondsmoothing process is set for a shifted synthesized image I_(S)(j, i).Next, a process of applying a two-dimensional low-pass filter{F_(LPF)(j_(LPF), i_(LPF))|−n_(LPF)≤j_(LPF)≤n_(LPF),−m_(LPF)≤i_(LPF)≤m_(LPF)} to the shifted synthesized image I_(S)(j, i)is executed. According to Formula (17), an unsharpness mask I_(USM)(j,i) is calculated. It is possible to use a two-dimensional filter suchas, for example, ^(t)[1, 0, 2, 0, 1]×[1, 0, 2, 0, 1], in thetwo-dimensional low-pass filter F_(LPF) (j_(LPF), i_(LPF)). Whennecessary, a two-dimensional Gaussian distribution or the like may beused.

$\begin{matrix}{{I_{USM}\left( {j,i} \right)} = {{I_{S}\left( {j,i} \right)} - {\sum\limits_{j_{LPF} = {- n_{LPF}}}^{n_{LPF}}{\sum\limits_{i_{LPF} = {- m_{LPF}}}^{m_{LPF}}{{F_{LPF}\left( {j_{LPF},i_{LPF}} \right)} \times {{I_{S}\left( {{j + j_{LPF}},{i + i_{LPF}}} \right)}.}}}}}} & (17)\end{matrix}$

In step S8, finally, the second sharpening or smoothing process isperformed. The refocus image I_(RF)(j, i) that is an output image isgenerated by applying the unsharpness mask I_(USM)(j, i) according tothe image shift difference amount distribution M_(DIFF)(j, i) accordingto Formula (18) to the shifted synthesized image I_(S)(j, i).I _(RF)(j,i)=I _(S)(j,i)+k _(USM) ×M _(DIFF)(j,i)×I _(USM)(j,i).  (18)

Formula (18) indicates the following process in an area in which theimage shift difference amount distribution is greater than or equal to 0(M_(DIFF)(j, i)≥0). That is, the process is a (second sharpening)process of sharpening the shifted synthesized image I_(S)(j, i)according to a magnitude of the image shift difference amountdistribution M_(DIFF)(j, i) using the unsharpness mask I_(USM)(j, i)multiplied by a positive coefficient k_(USM)×M_(DIFF)(j, i).

On the other hand, Formula (18) indicates the following process in anarea in which the image shift difference amount distribution is lessthan 0 (M_(DIFF)(j, i)≤0). That is, the process is a (second smoothing)process of smoothing the shifted synthesized image I_(S)(j, according toa magnitude of the image shift difference amount distributionM_(DIFF)(j, i) using the unsharpness mask I_(USM)(j, i) multiplied by anegative coefficient k_(USM)×M_(DIFF)(j, i).

In the refocusing in the shift synthesis process, it is possible toperform refocusing based on an optical principle using LF data. Therefocusing in the shift synthesis process is advantageous in that theprocess is performed even in an area in which the image shift differenceamount distribution is not detectable. The following case is present,however, when the pupil division direction is only one direction of thex-axis direction (the y-axis direction) as in the pupil division of thepresent embodiment (N_(x)=2, N_(y)=1, and N_(LF)=2). That is, therefocus effect is obtained in the x-axis direction (the y-axisdirection) of the pupil division direction, but the refocus effect maynot be sufficiently obtained in the y-axis direction (the x-axisdirection) orthogonal to the pupil division direction. On the otherhand, in control of the blur due to sharpening and smoothing accordingto the image shift difference amount distribution, it is possible toobtain the refocus effect regardless of the pupil division direction.Therefore, in the present invention, a refocus process in whichrefocusing in the shift synthesis process and control of a blur insharpening and smoothing according to the image shift difference amountdistribution are combined is performed. Thereby, it is possible toobtain the refocus effect even in the direction orthogonal to the pupildivision direction.

In the above-described present embodiment, image processing ofsharpening and smoothing according to a contrast distribution and animage shift amount distribution is performed on the synthesized imageI_(S)(j, i) of the plurality of corrected viewpoint images and an outputimage is generated. When necessary, processes of steps S5 to S7 of FIG.7 that are refocusing in the shift synthesis process may be omitted,image processing of sharpening and smoothing according to the contrastdistribution and the image shift amount distribution is performed on thecaptured image I(j, i), and an output image may be generated. The imageprocessing according to the contrast distribution and the image shiftamount distribution may be any one of a sharpening process, a smoothingprocess, and a combination thereof when necessary.

In the present embodiment, image processing is strongly performed on thefollowing areas according to Formula (7A), Formula (7B), Formula (9),Formula (17), and Formula (18). That is, the image processing, such assharpening or smoothing, on the synthesized image (or the capturedimage) of the plurality of corrected viewpoint images is more stronglyperformed for each viewpoint image in an area in which a differencebetween contrasts is small than in an area in which the differencebetween the contrasts is large. Also, the image processing, such assharpening or smoothing, on the synthesized image (or the capturedimage) of the plurality of corrected viewpoint images is more stronglyperformed in an area in which a contrast distribution is large than inan area in which the contrast distribution is small.

In the present embodiment, according to Formula (9), Formula (17), andFormula (18), a sharpening process is performed in an area in which adifference from the predetermined shift amount (reference) of the imageshift amount distribution is small and a smoothing process is performedin an area in which the difference is large. According to Formula (9),Formula (17), and Formula (18), the sharpening process is more stronglyperformed in an area in which a difference from a predetermined shiftamount of the image shift amount distribution is small than in an areain which the difference is large. Also, according to Formula (9),Formula (17), and Formula (18), the smoothing process is more stronglyperformed in an area in which a difference from a predetermined shiftamount of the image shift amount distribution is large than in an areain which the difference is small. Image Processing of Viewpoint Changeand Blur Fogging Correction

Next, a second process to be performed in the second processing unit 103in the present embodiment will be described. The second processing unit103 generates an image of a virtual viewpoint by synthesizing viewpointimages and detects an amount of ghost (a ghost component) to reduce aghost (unnecessary light) occurring in the synthesized image. In thesecond process, image correction to be performed by a change of asynthesis ratio is considered among processes on the plurality ofviewpoint images. For example, there are processes such as a viewpointchange process and image blur correction. Hereinafter, an example of aprocess of correcting foreground blur fogging and reducing a degreethereof will be specifically described. There is a phenomenon in whichthe main object is hidden if a blur of a foreground (a second object)located in front of the main object (a first object) is large. Forexample, the photographer may align a focus on the main object among aplurality of objects and perform photographing for effectivelyemphasizing the main object by intentionally blurring the foreground orthe background. FIG. 22 illustrates an example of an image in which aforeground blur is applied to the main object against the intention ofthe photographer by blurring the foreground. In an area 800 of FIG. 22,a foreground (flowers) located on the side in front (the near side) ofthe main object (a bird) is significantly blurred. Thus, the foregroundblur fogging in which the main object is hidden occurs. To prevent thequality of a captured image from being degraded, the CPU 170 determinesa synthesis ratio when a plurality of viewpoint images are synthesizedso that the foreground blur does not affect an image of an objectintended by the user in a process of reducing the foreground blurfogging.

The reduction of a degree of foreground blur fogging for the main objectis executed by the image processing unit 130 according to an instructionof the CPU 170. The image processing unit 130 performs a process to bedescribed below by acquiring a plurality of viewpoint images acquired bythe imaging element 110. By setting j and i as integer variables, aposition that is j^(th) in the row direction and is i^(th) in the columndirection in the first and second viewpoint images is denoted by (j, i).A pixel of the position (j, i) in the first viewpoint image A is denotedby A(j, i) and a pixel of the position (j, i) in the second viewpointimage B is denoted by B(j, i).

A first step is a process of setting a predetermined area (denoted by R)for reducing a degree of foreground blur fogging for the main object andits boundary width (denoted by σ) and calculating a table function(denoted by T(j, i)). The CPU 170 sets the boundary width σ of thepredetermined area R by setting a predetermined area R=[j₁, j₂]×[i₁,i₂]. The CPU 170 calculates the table function T(j, i) according to thepredetermined area R and its boundary width σ according to theabove-described Formula (12).

A second step is a process of calculating weighting coefficients for thefirst and second viewpoint images. The CPU 170 calculates a firstweighting coefficient W_(a)(j, i) of the pixel A(j, i) according toFormula (19A) by setting a real coefficient w (−1≤w≤1). Likewise, theCPU 170 calculates a second weighting coefficient W_(b)(j, i) of thepixel B(j, i) according to Formula (19B).W _(a)(j,i)=1−wT(j,i)  (19A)W _(b)(j,i)=1+wT(j,i)  (19B)

A third step is an image generation process using a weightingcoefficient. The image processing unit 130 generates an output I(j, i)according to Formula (20) from the pixels A(j, i) and B(j, i) and theweighting coefficients W_(a)(j, i) and W_(b)(j, i).I(j,i)=W _(a)(j,i)*A(j,i)+W _(b)(j,i)*B(j,i)  (20)

When necessary, the image processing unit 130 generates an output imageI_(s)(j, i) according to Formula (21A) or (21B) in combination with arefocus process based on a shift amount s. Also, according to thedenotation of Formula (14) described above, the following A(j, i) andB(j, i) correspond to MI₁(j, i) and MI₂(j, i), respectively.I _(s)(j,i)=W _(a)(j,i)*A(j,i)+W _(b)(j,i)*B(j,i+s)  (21A)I _(s)(j,i)=W _(a)(j,i)*A(j,i)+W _(b)(j,i+s)*B(j,i+s)  (21B)

Next, a principle of image processing for reducing a degree offoreground blur for the main object will be described with reference toFIGS. 13A to 13C. The imaging element 110 is arranged on the imagingplane 600, and the exit pupil of the image forming optical system isdivided into the two pupil part areas 401 and 402. FIG. 13A is anoptical path diagram illustrating an example of a foreground blurfogging image for the main object. FIG. 13A illustrates a state in whicha blurred image (Γ1+Γ2) of the front-side object q2 is captured tooverlap the image p1 (a focus image) of the main object q1. FIGS. 13Band 13C illustrate optical path diagrams in which a light beam passingthrough the pupil part area 401 and a light beam passing through thepupil part area 402 are separated with respect to the state illustratedin FIG. 13A. In each pixel of the imaging element 110, a first viewpointimage is generated from a light reception signal of the firstphotoelectric conversion unit 301, and a second viewpoint image isgenerated from a light reception signal of the second photoelectricconversion unit 302.

In FIG. 13B, a light beam from the main object q1 passes through thepupil part area 401 and image formation is performed in an image p1 in afocused state, a light beam from the front-side object q2 passes throughthe pupil part area 401 and spreads to a blurred image Γ1 in a defocusedstate, and light is received in the photoelectric conversion unit ofeach pixel of the imaging element 110. In the first viewpoint image, theimage p1 and the blurred image Γ1 are captured without overlapping eachother. In this case, in a predetermined area (near the image p1 of theobject q1), an object (the blurred image Γ1 of the object q2) of thenear side is photographed in a narrowest range in the first and secondviewpoint images. Also, in a predetermined area (near the image p1 ofthe object q1), the appearance of the blurred image Γ1 of the object q2is less and a contrast evaluation value is largest in the first andsecond viewpoint images.

On the other hand, in FIG. 13C, a light beam from the main object q1passes through the pupil part area 402 and image formation is performedin the image p1 in a focused state, a light beam from the front-sideobject q2 passes through the pupil part area 402 and spreads to ablurred image Γ2 in a defocused state, and light is received in thephotoelectric conversion unit of each pixel of the imaging element 110.In the second viewpoint image, the image p1 and the blurred image Γ2 arecaptured without overlapping each other. In a predetermined area (nearthe image p1 of the object q1), an object (the blurred image Γ2 of theobject q2) of the near side is photographed in a widest range in thefirst and second viewpoint images. Also, in a predetermined area (nearthe image p1 of the object q1), the appearance of the blurred image Γ2of the object q2 is large, and a contrast evaluation value is smallestin the first and second viewpoint images.

In a predetermined area (near the image p1), the first weightingcoefficient W_(a) for the first viewpoint image, in which the overlapbetween the image p1 and the blurred image Γ1 is small, is set to begreater than the second weighting coefficient W_(b) for the secondviewpoint image in which the overlap between the image p1 and theblurred image Γ2 is large. For example, in a predetermined area of theoutput image, a weighting coefficient value of the viewpoint image, inwhich the object of the near side is captured in the widest range amongthe plurality of viewpoint images, is smallest, or a weightingcoefficient value of the viewpoint image, in which the object of thenear side is captured in the narrowest range, is largest. Also, in apredetermined area of the output image, the weighting coefficient valueof the viewpoint image, in which the contrast evaluation value issmallest, is smallest, or the weighting coefficient value of theviewpoint image, in which the contrast evaluation value is largest, islargest.

It is possible to generate an image in which the foreground blur foggingfor the main object is reduced by generating a synthesized output imageusing a weighting coefficient. A reduction effect of the foreground blurfogging by the blur adjustment process will be described with referenceto FIGS. 23A and 23B. FIGS. 23A and 23B illustrate an image examplebefore the blur adjustment process. Foreground blur fogging of a secondobject (flowers) for a first object (a bird) in a predetermined area1000 occurs. FIG. 23B illustrates an image example after the bluradjustment process. As indicated in the predetermined area 1000 within acircular frame of the dotted line in the image of FIG. 23A, the beak,eyes, and wings of the bird are covered with the foreground blur of theflowers in white. On the other hand, in the image of FIG. 23B, thisforeground blur is reduced. Because a blur shape is not changed outsidea predetermined area in which no blur adjustment process is performedwhen necessary, a weighting coefficient value is generally equally addedfor each of the plurality of viewpoint images and an output image isgenerated.

Next, a ghost reduction process to be performed by the second processingunit 103 and the synthesis unit 105 will be described. The secondprocessing unit 103 performs a process of determining a ghost(unnecessary light) and a ghost reduction process of reducing orremoving the ghost.

In the present embodiment, a process of calculating unnecessarycomponents in the viewpoint images and synthesizing the unnecessarycomponents of viewpoint images in a similar process in accordance with aprocess of synthesizing the viewpoint images of an output image isperformed, and the unnecessary components are subtracted from asynthesized image.

FIG. 24 illustrates a flowchart of a process of determining anunnecessary component (a ghost component) in the present embodiment. Thefollowing steps are mainly executed according to an image processingprogram serving as a computer program by the CPU 170 or the imageprocessing unit 130, or an execution instruction is output to each part.The following process starts, for example, when the imaging element 110captures an image (for example, during a mode in which sequentiallyimaged digital signals are output or during recording immediately afterimaging), or when image data from the memory is read to a temporarystorage area within the image processing unit 130.

First, in step S2401, the CPU 170 controls an imaging unit (an imagingsystem) constituted of the photographing lens 230, the imaging element110, and the A/D converter 120, such that the imaging unit images anobject and acquires an input image (a captured image). Alternatively,image data captured in advance and recorded on the image recordingmedium 107 is read to a temporary storage area within the imageprocessing unit 130, so that the input image is acquired. In the presentembodiment, a synthesized image obtained by synthesizing a plurality ofviewpoint images corresponding to light beams passing through differentpupil areas of the photographing lens 230 within the imaging element 110as the input image and a viewpoint image corresponding to a partialpupil area before synthesis is acquired as the input image. The presentinvention is not limited to the synthesized image and the viewpointimage as the input image, and each of the plurality of viewpoint imagesmay be acquired as the input image.

In step S2402, the CPU 170 controls the image processing unit 130 andcauses a pair of viewpoint images to be generated from the synthesizedimage and the viewpoint image. Specifically, it is possible to calculatea plurality of viewpoint images by taking a difference. Here, the imageprocessing unit 130 may execute some of various types of imageprocessing as described above when the viewpoint image is generated. Ifan input image is acquired in the form of a plurality of viewpointimages in step S2401, only some of various types of image processing maybe performed in the present step.

Next, in step S2403, the image processing unit 130 obtains relativedifference information of a pair of viewpoint images by taking adifference between the viewpoint images. Here, in the presentembodiment, a process of rounding a negative value in the relativedifference information down to a value of 0 to simplify an unnecessarycomponent reduction process to be described below is performed. Thus, anunnecessary component is detected only as a positive value.

Also, a process of aligning a pair of viewpoint images may be executedto remove the object viewpoint difference component when the relativedifference information is obtained in an image including a short-rangeobject. Specifically, it is possible to perform image alignment bydetermining a shift position at which a correlation between images ismaximized, while relatively shifting a position of the other image withrespect to one image between the pair of viewpoint images. Also, theimage alignment may be performed by determining a shift position atwhich a sum of the squares of the differences between the viewpointimages is minimized. Also, a focused area of the viewpoint image may bea target for determining a shift position for alignment.

Also, edge detection may be performed in each viewpoint image inadvance, and a shift position for alignment using an image indicatingthe detected edge may be determined. According to this method, it isdifficult to detect an edge having a high contrast in the focused areaand to detect an edge having a low contrast in a non-focused area, suchas a background. Thus, a shift position at which the focused area isinevitably emphasized is determined. Also, when the relative differenceimage is generated, a step, such as threshold value processing, may beadded to remove an influence, such as noise.

Next, in step S2404, a process of determining a component remaining inthe relative difference image generated in step S2403 as an unnecessarycomponent is executed. In step S2405, the image processing unit 130 addsunnecessary components of viewpoint images determined in step S2404 (asynthesis value of the unnecessary components is calculated).

Next, in step S2406, the image processing unit 130 performs a correctionprocess of reducing or removing a noise component from an unnecessarycomponent. Specifically, a process of subtracting noise included in anunnecessary component of each viewpoint image from the synthesis valueof unnecessary components calculated in step S2405 is executed.

Here, a procedure of the correction process of reducing or removing anoise component from a ghost component in step S2406 will be described.First, a noise component is calculated on the basis of a standarddeviation of a noise component (noise information) of the imagingelement 110 measured in advance. Here, a predicted value of the noisecomponent is measured from a result of pre-imaging an object of uniformluminance by the imaging element 110 and obtained for every ISOsensitivity having a large influence on noise, and is tabled. Thepresent measurement for each of the plurality of viewpoint images istime-consuming and is affected by shading for each viewpoint image.Therefore, in the present embodiment, the noise component is determinedfrom data measured in the above-described synthesized image in which aplurality of viewpoints corresponding to light beams from differentpupil areas of an optical system are synthesized. Also, all pixels forevery ISO may have a uniform noise component on the basis of a measuredvalue as a noise component, and each pixel may uniformly have the noisecomponent at each image height. A process of subtracting the calculatednoise component from the calculated synthesis value of the unnecessarycomponents is performed. At this time, because a noise componentincluded in an unnecessary component of each viewpoint image is addedfor each process of adding unnecessary components calculated in stepS2405, it is necessary to perform the process of subtracting the noisecomponent (the number of viewpoint images −1) times. A method ofsubtracting the noise component is not limited thereto. For example, thestandard deviation of the noise component of each viewpoint image may becalculated from an image. At this time, specifically, a process ofdividing an image into (10×10) local areas, calculating the standarddeviation of a pixel value within each area, and subtracting the noisecomponent in each area is performed.

Next, in step S2407, the image processing unit 130 performs a correctionprocess of reducing or removing an unnecessary component from an imageto be output. Specifically, a process of subtracting the unnecessarycomponent calculated in step S2405 from the synthesized image acquiredin step S2401 is performed. Here, in the case of an embodiment in whichonly a plurality of viewpoint images are acquired without acquiring asynthesized image in step S2401, a corrected image is generated bysubtracting the unnecessary component calculated in step S2405 from thesynthesized image generated by synthesizing a plurality of viewpointimages. In step S2408, the image processing unit 130 performs a generalprocess on the corrected image and generates an output image to beoutput to the recording medium 160 or the image display unit 220. Atthis time, in addition to a developing process such as a normal whitebalance process or gamma correction, a well-known noise reductionprocess is also performed on a corrected image. In this process, it ispossible to reduce noise on the corrected image.

Finally, in step S2409, the CPU 170 performs control such that an outputimage from which an unnecessary component is removed or reduced isrecorded on the recording medium 160. Alternatively, a process ofdisplaying the output image on the image display unit ends.

As described above, it is possible to implement a good process ofreducing the number of unnecessary components from an image by reducingor removing a noise component from the unnecessary component in theimage processing device that reduces the number of unnecessarycomponents due to unnecessary light or the like from an image based on aplurality of viewpoint images.

In the present embodiment, an example of a synthesized image previouslysubjected to analog synthesis within an imaging sensor at the time of anoutput from the sensor, or a synthesized image obtained by synthesizinga plurality of viewpoint images is shown using a target image of theghost reduction process as an image based on a plurality of viewpointimages. The target image of the process is not, however, limitedthereto. For example, a corresponding unnecessary component may becalculated from any viewpoint image, or the like, and a reductionprocess may be performed.

FIG. 25 is a flowchart illustrating an operation of the image processingdevice in the present embodiment. The following process is executed bythe CPU 170 or each part according to an instruction of the CPU 170. Inthe present embodiment, an image processing device in which a pluralityof distinctive image processings using a plurality of viewpoint imagesare able to be applied to an image is implemented.

In step S2501, data of a plurality of viewpoint images is acquired froma photoelectrically converted signal after the first and secondphotoelectric conversion units of each pixel of the imaging element 110receive light. An input of the plurality of viewpoint images is notlimited to an image immediately after imaging, and it is possible toacquire an image file via a memory card or a network. Also, there is amethod of applying a set value pre-maintained by the image processingdevice as a method of designating an adjustment value according to auser operation using a user interface (UI) unit.

In step S2502, the CPU 170 acquires the above-described adjustment valueof at least one of a process of adjusting a perceived resolution, aviewpoint change process, or a blur fogging correction process, and aghost reduction process for a viewpoint image. The CPU 170 acquires thepresence/absence of application of the process of adjusting theperceived resolution input from the operation unit 210 by the user,adjustment values of strengths of first sharpening and smoothingprocesses and second sharpening and smoothing processes, and anadjustment value of a shift amount for shift synthesis. Also, the CPU170 acquires the presence/absence of application of the viewpoint changeprocess input from the operation unit 210 by the user, positioninformation of an area within an image for which a viewpoint is changed,and an adjustment value indicating an adjusted position of a viewpoint.Also, the CPU 170 acquires the presence/absence of application of theghost reduction process input from the operation unit 210 by the user,position information of an area for application thereto, and anadjustment value related to the strength of the process.

In the present embodiment, types of image and image processing and anadjustment value are set on an application provided by the imageprocessing device for a synthesized image (a captured image) of aplurality of viewpoint images obtained from the imaging element 110.Thereby, it is possible to perform the process of adjusting theperceived resolution, the viewpoint change process, and the ghostreduction process on the synthesized image. Further, in the presentembodiment, it is possible to apply the image processing to the sameimage.

Here, if there is a plurality of viewpoint images corresponding to pupildivision areas, as in the present embodiment as described above, thegradual pupil division is performed. Thus, it is desirable to apply thefirst sharpening and smoothing processes for further improving theeffect of the adjustment process to the viewpoint image for use in theprocess of adjusting the perceived resolution. When the smooth viewpointchange is made in the gradual pupil division for the ghost reductionprocess or the viewpoint change process, however, a synthesis result inwhich it is difficult to view a joint is obtained. Therefore, in thepresent embodiment, an influence of the first sharpening and smoothingprocesses on another process is reduced by weakly setting the strengthsof the first sharpening and smoothing processes to be performed when theprocess of adjusting the perceived resolution is applied as comparedwith when the other process is not applied. Of course, the case in whichthe strengths of the first sharpening and smoothing processes are weaklyset also includes the case in which the first sharpening and smoothingprocesses are not performed.

Also, because a technique of subtracting the ghost component calculatedfrom each viewpoint image from the synthesized image is taken, it ismore preferable to apply the ghost reduction process finally afteranother process ends. In view of the above description, each imageprocessing is performed in the following flow.

In step S2503, the first sharpening and smoothing processes areperformed if the process of adjusting the perceived resolution is set.The CPU 170 performs the first sharpening and smoothing processes on aplurality of viewpoint images on the basis of adjustment values ofstrengths of the first sharpening and smoothing processes acquired onthe basis of an operation input to the operation unit 210.

In step S2504, a process of synthesizing a plurality of viewpoint imagesis performed on the basis of settings of a relative shift of a pluralityof viewpoint images according to the process of adjusting the perceivedresolution and a synthesis ratio according to the viewpoint changeprocess. If the first sharpening and smoothing processes are performedon the viewpoint image, the synthesis process is performed on theplurality of viewpoint images after the application on the basis of ashift based on the adjustment value and a synthesis ratio based onposition information of an area for application thereto and informationof an adjustment value of a viewpoint as in Formula (21A) or Formula(21B). Also, if the ghost reduction process is performed thereafter,each viewpoint image immediately before synthesis is assumed to bestored in the memory.

In step S2505, the second sharpening and smoothing processes related tothe process of adjusting the perceived resolution are performed on animage after synthesis. Here, the corresponding second sharpening andsmoothing processes are also performed on the viewpoint imageimmediately before synthesis stored in the memory. In step S2506, theabove-described ghost reduction process is performed using thesynthesized image after the second sharpening and smoothing processesand each viewpoint image.

In step S2507, a display process or a recording process on the synthesisresult is performed. A synthesized image obtained as a result ofprocessing up to S2506 is displayed on a display device, such as theimage display unit 220, and/or recorded in a recording unit, such as therecording medium 160, or the like, and a series of operations end. Ifthe user views the displayed synthesized image after image processingand changes the image processing (the adjustment value) again, theprocess of the present flowchart is executed from the beginning again.Also, in addition to the image display in the output form of thesynthesized image in step S2507, there is a form in which thesynthesized image is recorded as an image file in a recording mediumsuch as a memory card. Also, there is a form in which an image filesaved in the recording medium is transmitted to an external device via anetwork or the like.

As described above, according to the present embodiment, an imageprocessing device capable of appropriately performing a plurality ofdistinctive synthesis processes using a plurality of viewpoint images isimplemented. It is possible to selectively apply a plurality ofdifferent image processings to the image data and to simultaneouslyapply a plurality of adjustment values determined for adjustment itemsfor the viewpoint image.

Second Embodiment

Next, the second embodiment of the present invention will be described.The configuration of the image processing device and the description ofeach image processing related to FIGS. 1 to 25 are similar to those ofthe first embodiment. Components of the present embodiment similar tothose of the first embodiment use already-used reference signals, anddetailed description thereof is omitted. Therefore, differences betweenthe embodiments will be mainly described. This omission of thedescription is true for the following embodiment.

The control unit 104 of FIG. 1 has a plurality of control modes and onlya first process is executed if a first condition is satisfied in a firstcontrol mode. That is, only a first processing unit 102 executes aprocess only in the case of a specific condition being met.Alternatively, if a second processing unit 103 is not included in theinstallation of the device, only a process of the first processing unit102 is executed. The first condition is, for example, as describedbelow.

If Effect of Ghost Reduction Process is Small

The ghost reduction process is a process of reducing a ghost occurringunder the specific condition using a viewpoint image. According to aspecific combination of a camera and a lens, the ghost reduction effectmay be small. If a combination with the small effect is predetermined,the CPU 170 performs control such that the second process is notexecuted. That is, if an image captured in the condition of the smallghost reduction effect is input, control is performed such that thereception of the adjustment value related to the second process isprevented. The CPU 170 draws the user's attention by clearly displayingthat adjustment is unnecessary on a display screen of the image displayunit 220.

When Image Processing Device in which Program According to the PresentEmbodiment is Embedded is Specialized in Refocus Function

In a case in which the image processing device in which the programaccording to the present embodiment is embedded is specialized in arefocus function, only the first process is executed and the user canonly adjust refocusing. In the present embodiment, application of onlythe first process is effective under the first condition, and theprocessing load or the power consumption can be reduced without applyingthe second process.

Third Embodiment

Next, the third embodiment of the present invention will be described.The configuration of the image processing device and the description ofeach image processing related to FIGS. 1 to 25 are similar to those ofthe first embodiment. The control unit 104 of FIG. 1 has a plurality ofcontrol modes, and only a second process is executed if a secondcondition is satisfied in a second control mode. That is, only thesecond processing unit 103 executes a process only in the case of aspecific condition being met. Alternatively, when the first processingunit 102 is not included in the installation of the device, only aprocess of the second processing unit 103 is executed. The secondcondition is, for example, as described below.

If a Refocus Process is Unnecessary (Effect is Small)

The shift synthesis process of virtually changing a focus during theprocess of adjusting a perceived resolution is not problematic in termsof a defocus, and its application may be unnecessary (the effect may besmall). Thus, the refocus process can be regarded to be unnecessaryaccording to a state during photographing in combination with objectinformation, or the like, from face recognition, or the like.

If Image Processing Device in which Program According to PresentEmbodiment is Embedded is Specialized for Ghost Reduction Function

In this case, there is no room for performing the first process, andonly removal or reduction of an appearing ghost is possible. In thepresent embodiment, application of only the second process is effectiveunder the second condition, and the processing load or the powerconsumption can be reduced without applying the first process.

Both the first condition described in the second embodiment and thesecond condition described in the third embodiment may not be satisfied,and the control unit 104 of FIG. 1 also may perform control in whichfirst and second processes are not executed. For example, because thedegree of depth of the object is deep if the user squeezes the apertureduring photographing, the effect of the parallax is reduced. In extremecases, the effect of the parallax is absent, and the effect of therefocus process, the ghost reduction process, or the like using theparallax is lost. In the case of photographing in this photographingcondition, the control unit 104 presents message display or the like tothe user, and does not perform adjustment, so that processing of aviewpoint image is not applied. Also, noise may increase by improvingthe ISO sensitivity in addition to the aperture. In the case of an ISOsensitivity greater than or equal to a certain fixed level, the controlunit 104 does not perform adjustment similar to the case of theaperture. As described above, in the case of the specific conditionbeing met, the control unit 104 disables the first and second processeswithout applying the first and second processes. Thereby, it is possibleto achieve operational ease and securing of image quality or the like.

Fourth Embodiment

Next, the fourth embodiment of the present invention will be described.The configuration of the image processing device and the description ofeach image processing related to FIGS. 1 to 25 are similar to those ofthe first embodiment. It is characterized in that control is exclusivelyperformed using the fact that image processing using a plurality ofviewpoint images described in the first embodiment does not affect eachother. In the present embodiment, it is possible to exclusively select aplurality of image processings using a plurality of viewpoint images bythe operation unit 210 of FIG. 2, and information of set imageprocessing in step S2502 of FIG. 25 and information of its adjustmentvalue are acquired. In steps S2503 to S2506, only the selected imageprocessing and the process related to the adjustment value areperformed, and other processes and steps are ignored.

Here, a further additional advantage of exclusive control will bedescribed. If a refocus calculation is performed in the first process,the CPU 170 does not perform foreground blur fogging correction in thesecond process. The image processing unit 130 requires a shift amount sif an output image I_(S)(j, i) is generated according to Formula (4A) or(4B). Thus, processing is likely to be delayed because a predeterminedtime or more is required to generate the output image I_(S)(j, i) beforethe shift amount s is determined. In this case, the CPU 170 exclusivelycontrols the first process and the second process, and the shift amounts is predetermined if the foreground blur fogging correction isperformed in the second process. In this case, it is possible to reducethe processing load of the image processing unit 130 without executingthe first process. In the present embodiment, it is possible to performa plurality of types of image processing on a plurality of viewpointimages at a suitable timing by exclusively controlling the first processand the second process.

OTHER EMBODIMENTS

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions (e.g., one or more programs) recorded on a storage medium(that may also be referred to more fully as a ‘non-transitorycomputer-readable storage medium’) to perform the functions of one ormore of the above-described embodiments and/or that includes one or morecircuits (e.g., an application specific integrated circuit (ASIC)) forperforming the functions of one or more of the above-describedembodiments, and by a method performed by the computer of the system oran apparatus by, for example, reading out and executing the computerexecutable instructions from the storage medium to perform the functionsof one or more of the above-described embodiments and/or controlling theone or more circuits to perform the functions of one or more of theabove-described embodiments. The computer may comprise one or moreprocessors (e.g., a central processing unit (CPU), or a micro processingunit (MPU)) and may include a network of separate computers or separateprocessors to read out and to execute the computer executableinstructions. The computer executable instructions may be provided tothe computer, for example, from a network or the storage medium. Thestorage medium may include, for example, one or more of a hard disk, arandom-access memory (RAM), a read only memory (ROM), a storage ofdistributed computing systems, an optical disk (such as a compact disc(CD), a digital versatile disc (DVD), or a Blu-ray Disc (BD)™), a flashmemory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

What is claimed is:
 1. An image processing device comprising: (A) aplurality of image sensors configured to acquire a plurality ofviewpoint images; (B) a memory that stores instructions; and (C) atleast one processor coupled to the memory and configured to execute theinstructions: (a) to perform image processing on image data based on theplurality of viewpoint images acquired by the plurality of imagesensors, and to selectively apply, in performing image processing, aplurality of different image processings to the image data, theplurality of different image processings including: (i) an adjustmentprocess of adjusting a perceived resolution of an image, the adjustmentprocess including a shift synthesis process of relatively shifting theplurality of viewpoint images to synthesize the plurality of viewpointimages that are relatively shifted; (ii) a viewpoint change process ofchanging a viewpoint by changing a weighting coefficient when theplurality of viewpoint images are synthesized; and (iii) a ghostreduction process of generating an image in which an influence of aghost of the plurality of viewpoint images is reduced; (b) to set theimage processing to be applied in performing the image processing byexclusively setting one of the plurality of different image processingsand to set a parameter of the image processing to be applied to theimage data; and (c) to perform other image processing using theplurality of viewpoint images for which the ghost is reduced accordingto the ghost reduction process if the ghost reduction process is set andone of the adjustment process and the viewpoint change process isapplied.
 2. The image processing device according to claim 1, whereinthe at least one processor further executes the instructions (d) tooutput an image generated by synthesizing the plurality of viewpointimages according to the set image processing.
 3. The image processingdevice according to claim 1, wherein the at least one processor furtherexecutes the instructions (d) to perform at least one of a sharpeningprocess and a smoothing process on the plurality of viewpoint imagesbefore performing the one of the plurality of different imageprocessings.
 4. The image processing device according to claim 1,wherein the at least one processor further executes the instructions (d)to perform at least one of a sharpening process and a smoothing processon the plurality of viewpoint images before performing the one of theplurality of different image processings, wherein the sharpening processand the smoothing process are performed more weakly when one of theviewpoint change process and the ghost reduction process is set to beapplied with the adjustment process than when only the adjustmentprocess is set to be applied.
 5. The image processing device accordingto claim 1, wherein the at least one processor further executes theinstructions (d) to synthesize the plurality of viewpoint images after ashift at a synthesis ratio according to the shift synthesis processincluded in the viewpoint change process by relatively shifting theplurality of viewpoint images according to the adjustment process if theadjustment process and the viewpoint change process are set to beapplied.
 6. The image processing device according to claim 1, whereinthe adjustment process includes one of a process of performing asharpening process and a smoothing process on an image on the basis ofat least one of a distance from a focusing position of the image and acontrast.
 7. The image processing device according to claim 1, whereinthe ghost reduction process includes a process of subtracting a ghostcomponent obtained on the basis of a difference among the plurality ofviewpoint images from an image.
 8. The image processing device accordingto claim 1, wherein the plurality of image sensors includes a pluralityof photodiodes that obtain pixels signals by photoelectricallyconverting light passing through different pupil part areas of theplurality of image sensors, and the plurality of image sensors acquiresthe plurality of viewpoint images generated from the obtained pixelsignals.
 9. An imaging device comprising: (A) a plurality of imagesensors configured to acquire an image of an object; (B) a memory thatstores instructions; and (C) at least one processor coupled to thememory and configured to execute the instructions: (a) to perform imageprocessing on image data based on the plurality of viewpoint imagesacquired by the plurality of image sensors, and to selectively apply aplurality of different image processings to the image data, theplurality of different image processings including: (i) an adjustmentprocess of adjusting a perceived resolution of an image, the adjustmentprocess including a shift synthesis process of relatively shifting theplurality of viewpoint images to synthesize the plurality of viewpointimages that are relatively shifted; (ii) a viewpoint change process ofchanging a viewpoint by changing a weighting coefficient when theplurality of viewpoint images are synthesized; and (iii) a ghostreduction process of generating an image in which an influence of aghost of the plurality of viewpoint images is reduced; and (b) to setthe image processing to be applied in performing the image processing byexclusively setting one of the plurality of different image processingsand to seta parameter of the image processing to be applied to the imagedata; and (c) to perform other image processing using the plurality ofviewpoint images for which the ghost is reduced according to the ghostreduction process if the ghost reduction process is set and one of theadjustment process and the viewpoint change process is applied.
 10. Theimaging device according to claim 9, wherein each of the plurality ofimage sensors includes at least one microlens and a plurality ofphotodiodes, and wherein signals of the plurality of viewpoint imagesare generated from signals output from the plurality of photodiodescorresponding to the plurality of image sensors.
 11. The imaging deviceaccording to claim 10, wherein the plurality of different imageprocessings further includes (iv) a refocus calculation that calculatesa shift amount, and wherein the at least one processor further executesthe instructions (d) to detect a defocus amount from the shift amountcalculated in the performed refocus calculation.
 12. An image processingmethod comprising: (A) acquiring a plurality of viewpoint images; (B)performing image processing on image data, based on the plurality ofviewpoint images, by selectively applying one or more of a plurality ofdifferent image processings to the image data, the plurality of imageprocessings including: (a) an adjustment process of adjusting aperceived resolution of an image, the adjustment process including ashift synthesis process of relatively shifting the plurality ofviewpoint images to synthesize the plurality of viewpoint images thatare relatively shifted; (b) a viewpoint change process of changing aviewpoint by changing a weighting coefficient when the plurality ofviewpoint images are synthesized; and (c) a ghost reduction process ofgenerating an image in which an influence of a ghost of the plurality ofviewpoint images is reduced; (C) setting the image processing to beapplied in the step of performing the image processing, one of theplurality of different image processings being exclusively set, and aparameter of the image processing to be applied to the image data beingsettable in the setting; and (D) performing other image processing usingthe plurality of viewpoint images for which the ghost is reducedaccording to the ghost reduction process if the ghost reduction processis set and one of the adjustment process and the viewpoint changeprocess is applied.
 13. A non-transitory computer-readable recordingmedium storing a program for causing a computer to execute an imageprocessing method comprising: (A) acquiring a plurality of viewpointimages; (B) performing image processing on image data, based on theplurality of viewpoint images, by selectively applying one or more of aplurality of different image processings to the image data, theplurality of image processings including: (a) an adjustment process ofadjusting a perceived resolution of an image, the adjustment processincluding a shift synthesis process of relatively shifting the pluralityof viewpoint images to synthesize the plurality of viewpoint images thatare relatively shifted; (b) a viewpoint change process of changing aviewpoint by changing a weighting coefficient when the plurality ofviewpoint images are synthesized; and (c) a ghost reduction process ofgenerating an image in which an influence of a ghost of the plurality ofviewpoint images is reduced; (C) setting the image processing to beapplied in the step of performing the image processing one of theplurality of different image processings being set exclusively, and aparameter of the image processing to be applied to the image data beingsettable in the setting; and (D) performing other image processing usingthe plurality of viewpoint images for which the ghost is reducedaccording to the ghost reduction process if the ghost reduction processis set and one of the adjustment process and the viewpoint changeprocess is applied.